Apr 23 17:39:03.669353 ip-10-0-137-68 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Apr 23 17:39:03.669363 ip-10-0-137-68 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Apr 23 17:39:03.669370 ip-10-0-137-68 systemd[1]: kubelet.service: Failed with result 'resources'. Apr 23 17:39:03.669593 ip-10-0-137-68 systemd[1]: Failed to start Kubernetes Kubelet. Apr 23 17:39:13.802850 ip-10-0-137-68 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Apr 23 17:39:13.802866 ip-10-0-137-68 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7fd28bcc838647f9b0ff9247d5271e19 -- Apr 23 17:41:40.863884 ip-10-0-137-68 systemd[1]: Starting Kubernetes Kubelet... Apr 23 17:41:41.305719 ip-10-0-137-68 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 23 17:41:41.305719 ip-10-0-137-68 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 23 17:41:41.305719 ip-10-0-137-68 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 23 17:41:41.305719 ip-10-0-137-68 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 23 17:41:41.305719 ip-10-0-137-68 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 23 17:41:41.306749 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.306608 2575 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 23 17:41:41.312491 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312474 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 23 17:41:41.312491 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312491 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312495 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312499 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312502 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312506 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312509 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312512 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312515 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312518 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312521 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312524 2575 feature_gate.go:328] unrecognized feature gate: Example2 Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312527 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312530 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312532 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312535 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312538 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312540 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312543 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312546 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312549 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 23 17:41:41.312560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312551 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312559 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312563 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312567 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312570 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312573 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312576 2575 feature_gate.go:328] unrecognized feature gate: Example Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312579 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312583 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312599 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312602 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312605 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312607 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312610 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312613 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312615 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312618 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312621 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312623 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312626 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 23 17:41:41.313071 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312628 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312631 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312633 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312636 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312639 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312641 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312644 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312646 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312649 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312653 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312656 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312658 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312661 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312664 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312667 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312670 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312673 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312676 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312678 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 23 17:41:41.313597 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312681 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312684 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312686 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312689 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312691 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312694 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312696 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312700 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312704 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312707 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312709 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312712 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312714 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312717 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312721 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312723 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312726 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312729 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312731 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 23 17:41:41.314069 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312734 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312736 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312739 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312741 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312744 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312746 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.312749 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313163 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313169 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313172 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313175 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313178 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313180 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313183 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313186 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313188 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313191 2575 feature_gate.go:328] unrecognized feature gate: Example Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313194 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313196 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313199 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 23 17:41:41.314528 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313201 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313204 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313207 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313209 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313211 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313214 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313217 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313220 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313223 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313225 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313228 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313230 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313233 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313235 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313238 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313240 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313243 2575 feature_gate.go:328] unrecognized feature gate: Example2 Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313245 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313248 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313250 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 23 17:41:41.315039 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313254 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313257 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313259 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313262 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313264 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313267 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313270 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313272 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313275 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313278 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313280 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313282 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313285 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313287 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313290 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313292 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313295 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313297 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313300 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313303 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 23 17:41:41.315533 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313306 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313308 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313310 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313313 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313315 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313318 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313320 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313323 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313325 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313328 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313330 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313333 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313336 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313339 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313344 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313347 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313352 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313355 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313358 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 23 17:41:41.316050 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313361 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313365 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313367 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313370 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313372 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313375 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313377 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313380 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313383 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313385 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313387 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313390 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313392 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.313395 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313471 2575 flags.go:64] FLAG: --address="0.0.0.0" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313486 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313496 2575 flags.go:64] FLAG: --anonymous-auth="true" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313500 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313505 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313508 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313513 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 23 17:41:41.316522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313518 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313522 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313525 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313529 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313534 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313537 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313541 2575 flags.go:64] FLAG: --cgroup-root="" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313543 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313547 2575 flags.go:64] FLAG: --client-ca-file="" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313550 2575 flags.go:64] FLAG: --cloud-config="" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313553 2575 flags.go:64] FLAG: --cloud-provider="external" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313556 2575 flags.go:64] FLAG: --cluster-dns="[]" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313560 2575 flags.go:64] FLAG: --cluster-domain="" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313563 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313566 2575 flags.go:64] FLAG: --config-dir="" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313569 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313572 2575 flags.go:64] FLAG: --container-log-max-files="5" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313576 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313580 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313583 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313599 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313603 2575 flags.go:64] FLAG: --contention-profiling="false" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313606 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313610 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313613 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 23 17:41:41.317074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313616 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313621 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313625 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313628 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313631 2575 flags.go:64] FLAG: --enable-load-reader="false" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313634 2575 flags.go:64] FLAG: --enable-server="true" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313637 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313642 2575 flags.go:64] FLAG: --event-burst="100" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313645 2575 flags.go:64] FLAG: --event-qps="50" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313648 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313651 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313654 2575 flags.go:64] FLAG: --eviction-hard="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313659 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313662 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313665 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313668 2575 flags.go:64] FLAG: --eviction-soft="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313671 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313675 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313678 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313681 2575 flags.go:64] FLAG: --experimental-mounter-path="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313684 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313687 2575 flags.go:64] FLAG: --fail-swap-on="true" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313690 2575 flags.go:64] FLAG: --feature-gates="" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313694 2575 flags.go:64] FLAG: --file-check-frequency="20s" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313698 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 23 17:41:41.317721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313701 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313704 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313708 2575 flags.go:64] FLAG: --healthz-port="10248" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313711 2575 flags.go:64] FLAG: --help="false" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313714 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313717 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313720 2575 flags.go:64] FLAG: --http-check-frequency="20s" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313723 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313727 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313731 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313734 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313737 2575 flags.go:64] FLAG: --image-service-endpoint="" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313740 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313743 2575 flags.go:64] FLAG: --kube-api-burst="100" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313746 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313749 2575 flags.go:64] FLAG: --kube-api-qps="50" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313752 2575 flags.go:64] FLAG: --kube-reserved="" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313755 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313759 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313762 2575 flags.go:64] FLAG: --kubelet-cgroups="" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313765 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313768 2575 flags.go:64] FLAG: --lock-file="" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313771 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313774 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 23 17:41:41.318318 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313777 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313783 2575 flags.go:64] FLAG: --log-json-split-stream="false" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313786 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313789 2575 flags.go:64] FLAG: --log-text-split-stream="false" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313792 2575 flags.go:64] FLAG: --logging-format="text" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313794 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313798 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313801 2575 flags.go:64] FLAG: --manifest-url="" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313804 2575 flags.go:64] FLAG: --manifest-url-header="" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313808 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313811 2575 flags.go:64] FLAG: --max-open-files="1000000" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313816 2575 flags.go:64] FLAG: --max-pods="110" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313819 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313822 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313825 2575 flags.go:64] FLAG: --memory-manager-policy="None" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313828 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313831 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313834 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313838 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313845 2575 flags.go:64] FLAG: --node-status-max-images="50" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313848 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313852 2575 flags.go:64] FLAG: --oom-score-adj="-999" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313855 2575 flags.go:64] FLAG: --pod-cidr="" Apr 23 17:41:41.318977 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313858 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313864 2575 flags.go:64] FLAG: --pod-manifest-path="" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313866 2575 flags.go:64] FLAG: --pod-max-pids="-1" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313870 2575 flags.go:64] FLAG: --pods-per-core="0" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313873 2575 flags.go:64] FLAG: --port="10250" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313876 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313895 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0def1d699727356c9" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313899 2575 flags.go:64] FLAG: --qos-reserved="" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313903 2575 flags.go:64] FLAG: --read-only-port="10255" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313906 2575 flags.go:64] FLAG: --register-node="true" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313909 2575 flags.go:64] FLAG: --register-schedulable="true" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313912 2575 flags.go:64] FLAG: --register-with-taints="" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313916 2575 flags.go:64] FLAG: --registry-burst="10" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313920 2575 flags.go:64] FLAG: --registry-qps="5" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313923 2575 flags.go:64] FLAG: --reserved-cpus="" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313926 2575 flags.go:64] FLAG: --reserved-memory="" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313929 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313932 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313936 2575 flags.go:64] FLAG: --rotate-certificates="false" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313938 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313942 2575 flags.go:64] FLAG: --runonce="false" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313944 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313948 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313951 2575 flags.go:64] FLAG: --seccomp-default="false" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313954 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313957 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 23 17:41:41.319571 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313960 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313963 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313967 2575 flags.go:64] FLAG: --storage-driver-password="root" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313970 2575 flags.go:64] FLAG: --storage-driver-secure="false" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313973 2575 flags.go:64] FLAG: --storage-driver-table="stats" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313976 2575 flags.go:64] FLAG: --storage-driver-user="root" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313979 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313982 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313985 2575 flags.go:64] FLAG: --system-cgroups="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313988 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.313997 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314000 2575 flags.go:64] FLAG: --tls-cert-file="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314003 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314007 2575 flags.go:64] FLAG: --tls-min-version="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314010 2575 flags.go:64] FLAG: --tls-private-key-file="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314013 2575 flags.go:64] FLAG: --topology-manager-policy="none" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314016 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314019 2575 flags.go:64] FLAG: --topology-manager-scope="container" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314022 2575 flags.go:64] FLAG: --v="2" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314026 2575 flags.go:64] FLAG: --version="false" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314031 2575 flags.go:64] FLAG: --vmodule="" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314036 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314039 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314127 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314131 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 23 17:41:41.320830 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314134 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314137 2575 feature_gate.go:328] unrecognized feature gate: Example2 Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314140 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314142 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314145 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314148 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314150 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314153 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314155 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314158 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314162 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314164 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314167 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314170 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314172 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314175 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314177 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314182 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314185 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314187 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 23 17:41:41.321878 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314190 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314192 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314195 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314198 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314200 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314203 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314207 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314209 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314212 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314215 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314217 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314220 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314222 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314225 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314228 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314230 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314233 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314235 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314238 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314241 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 23 17:41:41.322758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314243 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314245 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314248 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314251 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314254 2575 feature_gate.go:328] unrecognized feature gate: Example Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314256 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314259 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314261 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314264 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314268 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314271 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314274 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314278 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314281 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314283 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314285 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314288 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314291 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314297 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 23 17:41:41.323349 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314300 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314304 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314308 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314311 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314313 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314316 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314319 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314321 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314324 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314327 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314329 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314332 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314335 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314337 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314340 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314342 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314345 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314348 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314350 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 23 17:41:41.324150 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314353 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314356 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314358 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314362 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314364 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.314367 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.314375 2575 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.321492 2575 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.321515 2575 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321607 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321616 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321621 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321626 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321630 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321635 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 23 17:41:41.324950 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321640 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321645 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321649 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321653 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321658 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321662 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321666 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321671 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321675 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321682 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321689 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321694 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321700 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321705 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321709 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321713 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321719 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321723 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321728 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 23 17:41:41.325499 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321732 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321736 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321743 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321748 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321752 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321757 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321763 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321768 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321772 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321777 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321781 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321785 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321789 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321794 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321798 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321802 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321807 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321811 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321815 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321819 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 23 17:41:41.326206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321823 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321828 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321832 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321836 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321841 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321845 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321849 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321853 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321858 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321862 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321866 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321870 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321874 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321878 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321882 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321886 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321890 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321894 2575 feature_gate.go:328] unrecognized feature gate: Example2 Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321899 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321904 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 23 17:41:41.326769 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321908 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321913 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321917 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321921 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321925 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321929 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321933 2575 feature_gate.go:328] unrecognized feature gate: Example Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321937 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321941 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321945 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321950 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321954 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321959 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321963 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321967 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321971 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321975 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321980 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321984 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321988 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 23 17:41:41.327292 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.321992 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.322000 2575 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322161 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322169 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322174 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322178 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322182 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322186 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322191 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322195 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322199 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322203 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322208 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322212 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322217 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 23 17:41:41.327812 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322221 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322225 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322229 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322233 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322237 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322242 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322246 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322250 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322254 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322258 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322262 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322266 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322270 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322275 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322279 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322284 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322288 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322292 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322297 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322301 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 23 17:41:41.328203 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322305 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322309 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322313 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322318 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322322 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322326 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322330 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322334 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322338 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322343 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322348 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322351 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322355 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322360 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322364 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322369 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322373 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322376 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322380 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 23 17:41:41.328709 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322385 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322389 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322393 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322397 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322401 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322405 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322409 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322413 2575 feature_gate.go:328] unrecognized feature gate: Example2 Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322417 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322421 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322426 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322432 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322436 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322440 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322444 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322449 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322453 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322457 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322461 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322465 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 23 17:41:41.329206 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322469 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322474 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322478 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322482 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322487 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322491 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322495 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322499 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322503 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322507 2575 feature_gate.go:328] unrecognized feature gate: Example Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322511 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322517 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322522 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:41.322527 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.322535 2575 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 23 17:41:41.329710 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.322718 2575 server.go:962] "Client rotation is on, will bootstrap in background" Apr 23 17:41:41.330100 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.325176 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 23 17:41:41.330100 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.326431 2575 server.go:1019] "Starting client certificate rotation" Apr 23 17:41:41.330100 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.326530 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 23 17:41:41.330100 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.327113 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 23 17:41:41.348199 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.348176 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 23 17:41:41.350004 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.349984 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 23 17:41:41.370221 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.370196 2575 log.go:25] "Validated CRI v1 runtime API" Apr 23 17:41:41.376770 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.376744 2575 log.go:25] "Validated CRI v1 image API" Apr 23 17:41:41.377822 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.377807 2575 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 23 17:41:41.381381 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.381358 2575 fs.go:135] Filesystem UUIDs: map[25d26b7a-92f3-4cf9-a17b-b97a6a7affab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8ca07b5d-d6bc-4cd7-851f-6f38696119fe:/dev/nvme0n1p3] Apr 23 17:41:41.381445 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.381379 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 23 17:41:41.383215 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.383198 2575 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 23 17:41:41.387308 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.387204 2575 manager.go:217] Machine: {Timestamp:2026-04-23 17:41:41.38535403 +0000 UTC m=+0.404807037 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100580 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa576a232279d03198e454c59ef80 SystemUUID:ec2aa576-a232-279d-0319-8e454c59ef80 BootID:7fd28bcc-8386-47f9-b0ff-9247d5271e19 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:f6:47:1a:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:f6:47:1a:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:0b:b6:60:8e:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 23 17:41:41.387308 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.387304 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 23 17:41:41.387415 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.387382 2575 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 23 17:41:41.388333 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.388306 2575 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 23 17:41:41.388490 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.388336 2575 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 23 17:41:41.388536 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.388500 2575 topology_manager.go:138] "Creating topology manager with none policy" Apr 23 17:41:41.388536 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.388508 2575 container_manager_linux.go:306] "Creating device plugin manager" Apr 23 17:41:41.388536 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.388524 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 23 17:41:41.389422 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.389412 2575 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 23 17:41:41.390773 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.390763 2575 state_mem.go:36] "Initialized new in-memory state store" Apr 23 17:41:41.390876 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.390867 2575 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 23 17:41:41.393583 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.393573 2575 kubelet.go:491] "Attempting to sync node with API server" Apr 23 17:41:41.393633 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.393599 2575 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 23 17:41:41.393633 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.393612 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 23 17:41:41.393633 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.393620 2575 kubelet.go:397] "Adding apiserver pod source" Apr 23 17:41:41.393633 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.393629 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 23 17:41:41.394658 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.394646 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 23 17:41:41.394704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.394665 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 23 17:41:41.397539 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.397522 2575 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 23 17:41:41.399302 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.399288 2575 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 23 17:41:41.400651 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400639 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400657 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400664 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400670 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400676 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400682 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400687 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400692 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 23 17:41:41.400697 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400700 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 23 17:41:41.400914 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400706 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 23 17:41:41.400914 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400715 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 23 17:41:41.400914 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.400724 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 23 17:41:41.401633 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.401621 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 23 17:41:41.401674 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.401635 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 23 17:41:41.402112 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.402090 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ln9f7" Apr 23 17:41:41.405622 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.405601 2575 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 23 17:41:41.405720 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.405655 2575 server.go:1295] "Started kubelet" Apr 23 17:41:41.405780 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.405733 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 23 17:41:41.405852 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.405816 2575 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 23 17:41:41.405903 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.405873 2575 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 23 17:41:41.406482 ip-10-0-137-68 systemd[1]: Started Kubernetes Kubelet. Apr 23 17:41:41.407556 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.407541 2575 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 23 17:41:41.408494 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.408480 2575 server.go:317] "Adding debug handlers to kubelet server" Apr 23 17:41:41.408756 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.408734 2575 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Apr 23 17:41:41.408854 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.408751 2575 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Apr 23 17:41:41.408933 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.408884 2575 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Apr 23 17:41:41.411534 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.411516 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 23 17:41:41.411947 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.411932 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 23 17:41:41.412798 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412779 2575 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 23 17:41:41.412798 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412778 2575 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 23 17:41:41.412930 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412809 2575 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 23 17:41:41.412930 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412922 2575 reconstruct.go:97] "Volume reconstruction finished" Apr 23 17:41:41.412930 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412930 2575 reconciler.go:26] "Reconciler: start to sync state" Apr 23 17:41:41.413065 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412935 2575 factory.go:55] Registering systemd factory Apr 23 17:41:41.413065 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.412969 2575 factory.go:223] Registration of the systemd container factory successfully Apr 23 17:41:41.413065 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.413011 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:41.413213 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.413195 2575 factory.go:153] Registering CRI-O factory Apr 23 17:41:41.413257 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.413219 2575 factory.go:223] Registration of the crio container factory successfully Apr 23 17:41:41.413305 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.413276 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 23 17:41:41.413305 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.413301 2575 factory.go:103] Registering Raw factory Apr 23 17:41:41.413383 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.413334 2575 manager.go:1196] Started watching for new ooms in manager Apr 23 17:41:41.414229 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.414010 2575 manager.go:319] Starting recovery of all containers Apr 23 17:41:41.415196 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.415175 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ln9f7" Apr 23 17:41:41.416825 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.415920 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-68.ec2.internal.18a90d412431544e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-68.ec2.internal,UID:ip-10-0-137-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-68.ec2.internal,},FirstTimestamp:2026-04-23 17:41:41.405619278 +0000 UTC m=+0.425072285,LastTimestamp:2026-04-23 17:41:41.405619278 +0000 UTC m=+0.425072285,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-68.ec2.internal,}" Apr 23 17:41:41.416997 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.416976 2575 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Apr 23 17:41:41.417144 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.417121 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Apr 23 17:41:41.423837 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.423771 2575 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 23 17:41:41.427854 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.427439 2575 manager.go:324] Recovery completed Apr 23 17:41:41.429118 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.429094 2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Apr 23 17:41:41.431976 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.431964 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 23 17:41:41.434160 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.434145 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientMemory" Apr 23 17:41:41.434240 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.434176 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasNoDiskPressure" Apr 23 17:41:41.434240 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.434190 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientPID" Apr 23 17:41:41.435400 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.435381 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 23 17:41:41.435400 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.435399 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 23 17:41:41.435537 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.435416 2575 state_mem.go:36] "Initialized new in-memory state store" Apr 23 17:41:41.438619 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.438604 2575 policy_none.go:49] "None policy: Start" Apr 23 17:41:41.438619 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.438620 2575 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 23 17:41:41.438711 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.438631 2575 state_mem.go:35] "Initializing new in-memory state store" Apr 23 17:41:41.477343 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477330 2575 manager.go:341] "Starting Device Plugin manager" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.477371 2575 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477385 2575 server.go:85] "Starting device plugin registration server" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477631 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477643 2575 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477740 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477810 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.477816 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.478279 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 23 17:41:41.499263 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.478313 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:41.578862 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.578774 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 23 17:41:41.579762 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.579743 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientMemory" Apr 23 17:41:41.579864 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.579779 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasNoDiskPressure" Apr 23 17:41:41.579864 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.579794 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientPID" Apr 23 17:41:41.579864 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.579822 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.588136 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.588115 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.588246 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.588145 2575 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-68.ec2.internal\": node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:41.589636 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.589617 2575 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 23 17:41:41.589721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.589646 2575 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 23 17:41:41.589721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.589660 2575 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 23 17:41:41.589721 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.589667 2575 kubelet.go:2451] "Starting kubelet main sync loop" Apr 23 17:41:41.589721 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.589699 2575 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 23 17:41:41.613924 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.613898 2575 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 23 17:41:41.643087 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.643056 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:41.690765 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.690726 2575 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal"] Apr 23 17:41:41.690944 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.690830 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 23 17:41:41.692305 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.692288 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientMemory" Apr 23 17:41:41.692372 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.692322 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasNoDiskPressure" Apr 23 17:41:41.692372 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.692332 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientPID" Apr 23 17:41:41.694748 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.694726 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 23 17:41:41.694883 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.694867 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.694930 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.694898 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 23 17:41:41.695488 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.695469 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientMemory" Apr 23 17:41:41.695577 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.695501 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasNoDiskPressure" Apr 23 17:41:41.695577 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.695514 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientPID" Apr 23 17:41:41.695577 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.695470 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientMemory" Apr 23 17:41:41.695577 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.695572 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasNoDiskPressure" Apr 23 17:41:41.695779 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.695600 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientPID" Apr 23 17:41:41.697793 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.697779 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.697858 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.697805 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 23 17:41:41.698446 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.698430 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientMemory" Apr 23 17:41:41.698527 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.698458 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasNoDiskPressure" Apr 23 17:41:41.698527 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.698472 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeHasSufficientPID" Apr 23 17:41:41.714581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.714559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0eff2e2d04276d2cb5575ebf2b719f08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal\" (UID: \"0eff2e2d04276d2cb5575ebf2b719f08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.714689 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.714601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eff2e2d04276d2cb5575ebf2b719f08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal\" (UID: \"0eff2e2d04276d2cb5575ebf2b719f08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.714689 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.714622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b84d58d2271f831c3e7ac872fa85f304-config\") pod \"kube-apiserver-proxy-ip-10-0-137-68.ec2.internal\" (UID: \"b84d58d2271f831c3e7ac872fa85f304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.717192 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.717175 2575 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-68.ec2.internal\" not found" node="ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.721469 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.721454 2575 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-68.ec2.internal\" not found" node="ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.743950 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.743929 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:41.815146 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.815119 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0eff2e2d04276d2cb5575ebf2b719f08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal\" (UID: \"0eff2e2d04276d2cb5575ebf2b719f08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.815262 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.815150 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eff2e2d04276d2cb5575ebf2b719f08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal\" (UID: \"0eff2e2d04276d2cb5575ebf2b719f08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.815262 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.815201 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b84d58d2271f831c3e7ac872fa85f304-config\") pod \"kube-apiserver-proxy-ip-10-0-137-68.ec2.internal\" (UID: \"b84d58d2271f831c3e7ac872fa85f304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.815262 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.815224 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b84d58d2271f831c3e7ac872fa85f304-config\") pod \"kube-apiserver-proxy-ip-10-0-137-68.ec2.internal\" (UID: \"b84d58d2271f831c3e7ac872fa85f304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.815262 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.815232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0eff2e2d04276d2cb5575ebf2b719f08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal\" (UID: \"0eff2e2d04276d2cb5575ebf2b719f08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.815397 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:41.815288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eff2e2d04276d2cb5575ebf2b719f08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal\" (UID: \"0eff2e2d04276d2cb5575ebf2b719f08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:41.844535 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.844483 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:41.945360 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:41.945319 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:42.018950 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.018926 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:42.023400 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.023379 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" Apr 23 17:41:42.046019 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:42.045986 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:42.146563 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:42.146476 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:42.247067 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:42.247018 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:42.326689 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.326645 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 23 17:41:42.327127 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.326828 2575 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 23 17:41:42.347819 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:42.347789 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-68.ec2.internal\" not found" Apr 23 17:41:42.385852 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.385828 2575 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 23 17:41:42.400917 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.400847 2575 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 23 17:41:42.411673 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.411637 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 23 17:41:42.412751 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.412732 2575 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" Apr 23 17:41:42.416544 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.416510 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-22 17:36:41 +0000 UTC" deadline="2027-10-10 12:26:20.379259794 +0000 UTC" Apr 23 17:41:42.416544 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.416540 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12834h44m37.962722514s" Apr 23 17:41:42.422961 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.422939 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 23 17:41:42.424099 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.424080 2575 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 23 17:41:42.424562 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.424549 2575 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" Apr 23 17:41:42.434688 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.434663 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 23 17:41:42.442730 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.442700 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-btcs4" Apr 23 17:41:42.449394 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.449372 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-btcs4" Apr 23 17:41:42.578019 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:42.577986 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff2e2d04276d2cb5575ebf2b719f08.slice/crio-a91db91ca2c9e9dbf84af6d6c5c4d7063cae923876fd6d6480ef906104072a54 WatchSource:0}: Error finding container a91db91ca2c9e9dbf84af6d6c5c4d7063cae923876fd6d6480ef906104072a54: Status 404 returned error can't find the container with id a91db91ca2c9e9dbf84af6d6c5c4d7063cae923876fd6d6480ef906104072a54 Apr 23 17:41:42.578424 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:42.578399 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84d58d2271f831c3e7ac872fa85f304.slice/crio-5387b4cef92d7d816b43e9ca5f946073315b6e5d5b1cc3f21241916f4589cac4 WatchSource:0}: Error finding container 5387b4cef92d7d816b43e9ca5f946073315b6e5d5b1cc3f21241916f4589cac4: Status 404 returned error can't find the container with id 5387b4cef92d7d816b43e9ca5f946073315b6e5d5b1cc3f21241916f4589cac4 Apr 23 17:41:42.582317 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.582301 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 17:41:42.592801 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.592750 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" event={"ID":"0eff2e2d04276d2cb5575ebf2b719f08","Type":"ContainerStarted","Data":"a91db91ca2c9e9dbf84af6d6c5c4d7063cae923876fd6d6480ef906104072a54"} Apr 23 17:41:42.593628 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.593601 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" event={"ID":"b84d58d2271f831c3e7ac872fa85f304","Type":"ContainerStarted","Data":"5387b4cef92d7d816b43e9ca5f946073315b6e5d5b1cc3f21241916f4589cac4"} Apr 23 17:41:42.916153 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:42.916120 2575 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 23 17:41:43.363075 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.363042 2575 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 23 17:41:43.395032 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.394997 2575 apiserver.go:52] "Watching apiserver" Apr 23 17:41:43.401980 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.401957 2575 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 23 17:41:43.402324 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.402302 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht","openshift-cluster-node-tuning-operator/tuned-rrrqf","openshift-dns/node-resolver-728bd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal","openshift-multus/multus-additional-cni-plugins-d7svv","openshift-network-operator/iptables-alerter-2fswp","kube-system/konnectivity-agent-bxxbs","openshift-image-registry/node-ca-xlkpg","openshift-multus/multus-lvm6f","openshift-multus/network-metrics-daemon-76492","openshift-network-diagnostics/network-check-target-28blj","openshift-ovn-kubernetes/ovnkube-node-p7h4b","kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal"] Apr 23 17:41:43.406780 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.406755 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:43.406901 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.406840 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:43.408869 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.408847 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.409016 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.408929 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.410781 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.410764 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.411403 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.411378 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t7mfq\"" Apr 23 17:41:43.411403 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.411397 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.411746 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.411719 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.411907 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.411889 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.411974 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.411940 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.413701 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.412583 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s95x9\"" Apr 23 17:41:43.414130 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.414107 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 23 17:41:43.414216 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.414154 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 23 17:41:43.414216 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.414212 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 23 17:41:43.414320 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.414281 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pm77z\"" Apr 23 17:41:43.415016 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.414793 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.415016 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.414842 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.417319 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.417072 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.418739 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.418719 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.418832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.418781 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 23 17:41:43.418984 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.418969 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.419042 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.418974 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-92cd5\"" Apr 23 17:41:43.419429 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.419270 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.420916 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.420898 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 23 17:41:43.421116 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.421100 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kbqfn\"" Apr 23 17:41:43.421330 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.421315 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.421405 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.421360 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 23 17:41:43.423090 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423071 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423577 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvc44\" (UniqueName: \"kubernetes.io/projected/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-kube-api-access-jvc44\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423633 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423658 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-kubernetes\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysctl-conf\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-lib-modules\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423622 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ktjg4\"" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/1870fd4e-15fe-4c77-a974-dd5e1934339a-kube-api-access-qpr8f\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-os-release\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysconfig\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysctl-d\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk96\" (UniqueName: \"kubernetes.io/projected/806976ba-94a9-41ca-b9b6-10181690a08a-kube-api-access-ctk96\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-systemd\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-run\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.423992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.423978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-var-lib-kubelet\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-host\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb810179-e96c-4f26-a7fc-e3006013faed-hosts-file\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krq4s\" (UniqueName: \"kubernetes.io/projected/7f9db399-293d-460a-8a3e-294b3c70c30e-kube-api-access-krq4s\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-modprobe-d\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhrq\" (UniqueName: \"kubernetes.io/projected/bb810179-e96c-4f26-a7fc-e3006013faed-kube-api-access-qwhrq\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1870fd4e-15fe-4c77-a974-dd5e1934339a-tmp\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-cnibin\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/806976ba-94a9-41ca-b9b6-10181690a08a-iptables-alerter-script\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/806976ba-94a9-41ca-b9b6-10181690a08a-host-slash\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-sys\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb810179-e96c-4f26-a7fc-e3006013faed-tmp-dir\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-tuned\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.424965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.424460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-system-cni-dir\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.425912 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.425894 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.426077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.426043 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:43.426165 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.426100 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:43.426566 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.426545 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cf5qv\"" Apr 23 17:41:43.426668 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.426581 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 23 17:41:43.427788 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.427769 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 23 17:41:43.427879 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.427785 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.427937 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.427894 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.428105 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.427998 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t6cwm\"" Apr 23 17:41:43.428430 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.428414 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.430249 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.430227 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 23 17:41:43.430328 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.430299 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 23 17:41:43.430378 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.430359 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 23 17:41:43.430676 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.430518 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 23 17:41:43.430922 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.430903 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 23 17:41:43.431104 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.431072 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wtfc9\"" Apr 23 17:41:43.431104 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.430972 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 23 17:41:43.450501 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.450440 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-22 17:36:42 +0000 UTC" deadline="2027-10-05 19:51:05.025132979 +0000 UTC" Apr 23 17:41:43.450501 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.450469 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12722h9m21.57466732s" Apr 23 17:41:43.513828 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.513804 2575 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 23 17:41:43.525654 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525626 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-run\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.525654 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525653 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-host\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525669 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95f1b7b2-68c2-4b1d-8939-d348b6b5daf0-agent-certs\") pod \"konnectivity-agent-bxxbs\" (UID: \"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0\") " pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-cni-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-cni-binary-copy\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525778 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-host\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.525824 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525804 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhrq\" (UniqueName: \"kubernetes.io/projected/bb810179-e96c-4f26-a7fc-e3006013faed-kube-api-access-qwhrq\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525834 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-run\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk925\" (UniqueName: \"kubernetes.io/projected/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-kube-api-access-bk925\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-cnibin\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-netns\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.525994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-run-netns\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-etc-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-cni-bin\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526090 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1870fd4e-15fe-4c77-a974-dd5e1934339a-tmp\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.526148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526115 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/806976ba-94a9-41ca-b9b6-10181690a08a-host-slash\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526151 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-socket-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526196 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/806976ba-94a9-41ca-b9b6-10181690a08a-host-slash\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-log-socket\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovn-node-metrics-cert\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526260 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb810179-e96c-4f26-a7fc-e3006013faed-tmp-dir\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-kubelet\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-slash\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526327 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.526336 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-node-log\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.526446 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:41:44.026406118 +0000 UTC m=+3.045859131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526483 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-tuned\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526506 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526530 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-system-cni-dir\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-daemon-config\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.526561 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526567 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb810179-e96c-4f26-a7fc-e3006013faed-tmp-dir\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-env-overrides\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526644 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-system-cni-dir\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovnkube-script-lib\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526711 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-lib-modules\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526738 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-os-release\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526773 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-host\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-serviceca\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-cni-bin\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526943 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysconfig\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.526991 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysconfig\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527013 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-lib-modules\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.527297 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527042 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-os-release\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527097 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk96\" (UniqueName: \"kubernetes.io/projected/806976ba-94a9-41ca-b9b6-10181690a08a-kube-api-access-ctk96\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-var-lib-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-ovn\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-k8s-cni-cncf-io\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-sys-fs\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527343 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-var-lib-kubelet\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527363 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb810179-e96c-4f26-a7fc-e3006013faed-hosts-file\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527390 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-krq4s\" (UniqueName: \"kubernetes.io/projected/7f9db399-293d-460a-8a3e-294b3c70c30e-kube-api-access-krq4s\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527451 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-var-lib-kubelet\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527442 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb810179-e96c-4f26-a7fc-e3006013faed-hosts-file\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-os-release\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-etc-kubernetes\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527547 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-modprobe-d\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-systemd-units\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527615 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-cnibin\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.528056 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527642 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527666 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/806976ba-94a9-41ca-b9b6-10181690a08a-iptables-alerter-script\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527685 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-modprobe-d\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-socket-dir-parent\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527705 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f9db399-293d-460a-8a3e-294b3c70c30e-cnibin\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-systemd\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527756 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-sys\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527786 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvc44\" (UniqueName: \"kubernetes.io/projected/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-kube-api-access-jvc44\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-system-cni-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-conf-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-multus-certs\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527889 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-sys\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.527955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgwj\" (UniqueName: \"kubernetes.io/projected/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-kube-api-access-rrgwj\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48sgp\" (UniqueName: \"kubernetes.io/projected/f3973fd7-657b-4408-87a3-48d5d15b2e21-kube-api-access-48sgp\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528065 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-kubernetes\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528092 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysctl-conf\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.528832 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528118 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/1870fd4e-15fe-4c77-a974-dd5e1934339a-kube-api-access-qpr8f\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528141 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-kubernetes\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528169 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528189 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/806976ba-94a9-41ca-b9b6-10181690a08a-iptables-alerter-script\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95f1b7b2-68c2-4b1d-8939-d348b6b5daf0-konnectivity-ca\") pod \"konnectivity-agent-bxxbs\" (UID: \"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0\") " pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-device-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528241 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysctl-conf\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovnkube-config\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528278 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysctl-d\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-cni-multus\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-registration-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-kubelet\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-hostroot\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mtjn\" (UniqueName: \"kubernetes.io/projected/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-kube-api-access-4mtjn\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528442 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-sysctl-d\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-cni-netd\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528487 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-systemd\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.529431 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528532 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-systemd\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.530053 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.528749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.530053 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.529086 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f9db399-293d-460a-8a3e-294b3c70c30e-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.530229 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.530211 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1870fd4e-15fe-4c77-a974-dd5e1934339a-tmp\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.530325 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.530267 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1870fd4e-15fe-4c77-a974-dd5e1934339a-etc-tuned\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.535078 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.535054 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhrq\" (UniqueName: \"kubernetes.io/projected/bb810179-e96c-4f26-a7fc-e3006013faed-kube-api-access-qwhrq\") pod \"node-resolver-728bd\" (UID: \"bb810179-e96c-4f26-a7fc-e3006013faed\") " pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.536139 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.536103 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krq4s\" (UniqueName: \"kubernetes.io/projected/7f9db399-293d-460a-8a3e-294b3c70c30e-kube-api-access-krq4s\") pod \"multus-additional-cni-plugins-d7svv\" (UID: \"7f9db399-293d-460a-8a3e-294b3c70c30e\") " pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.536349 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.536325 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/1870fd4e-15fe-4c77-a974-dd5e1934339a-kube-api-access-qpr8f\") pod \"tuned-rrrqf\" (UID: \"1870fd4e-15fe-4c77-a974-dd5e1934339a\") " pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.536565 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.536547 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk96\" (UniqueName: \"kubernetes.io/projected/806976ba-94a9-41ca-b9b6-10181690a08a-kube-api-access-ctk96\") pod \"iptables-alerter-2fswp\" (UID: \"806976ba-94a9-41ca-b9b6-10181690a08a\") " pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.537069 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.537048 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvc44\" (UniqueName: \"kubernetes.io/projected/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-kube-api-access-jvc44\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:43.629635 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629539 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-os-release\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629635 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629574 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-etc-kubernetes\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629635 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629610 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-systemd-units\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.629635 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629631 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-socket-dir-parent\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629655 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-systemd\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629682 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-system-cni-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629710 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-conf-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629716 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-os-release\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629733 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-multus-certs\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629677 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-etc-kubernetes\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629719 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-socket-dir-parent\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629762 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-conf-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629721 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-systemd\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629774 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-multus-certs\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629776 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-systemd-units\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629800 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-system-cni-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629816 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgwj\" (UniqueName: \"kubernetes.io/projected/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-kube-api-access-rrgwj\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629831 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629839 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48sgp\" (UniqueName: \"kubernetes.io/projected/f3973fd7-657b-4408-87a3-48d5d15b2e21-kube-api-access-48sgp\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629906 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95f1b7b2-68c2-4b1d-8939-d348b6b5daf0-konnectivity-ca\") pod \"konnectivity-agent-bxxbs\" (UID: \"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0\") " pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.629934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629934 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-device-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629959 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovnkube-config\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.629984 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-cni-multus\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630008 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-registration-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630030 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-kubelet\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630054 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-hostroot\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630060 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-cni-multus\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630077 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4mtjn\" (UniqueName: \"kubernetes.io/projected/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-kube-api-access-4mtjn\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630103 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-kubelet\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-device-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630103 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-cni-netd\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-hostroot\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-registration-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630147 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95f1b7b2-68c2-4b1d-8939-d348b6b5daf0-agent-certs\") pod \"konnectivity-agent-bxxbs\" (UID: \"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0\") " pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630193 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-cni-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630220 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-cni-binary-copy\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630247 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630293 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bk925\" (UniqueName: \"kubernetes.io/projected/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-kube-api-access-bk925\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.630563 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630319 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-cnibin\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630343 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-netns\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630367 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-run-netns\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630390 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-etc-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630413 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-cni-bin\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630440 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-cni-netd\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630453 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-socket-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630478 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-log-socket\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630511 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-cni-bin\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-run-netns\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95f1b7b2-68c2-4b1d-8939-d348b6b5daf0-konnectivity-ca\") pod \"konnectivity-agent-bxxbs\" (UID: \"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0\") " pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630542 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovnkube-config\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630565 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-log-socket\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-etc-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630505 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovn-node-metrics-cert\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-netns\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630636 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-kubelet\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630662 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-slash\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.631392 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-socket-dir\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-kubelet\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630711 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-slash\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-cni-dir\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630738 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-node-log\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630772 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-daemon-config\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630800 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630826 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-env-overrides\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630850 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovnkube-script-lib\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-cni-binary-copy\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630861 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630880 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-host\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630905 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-serviceca\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-node-log\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630930 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-cni-bin\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630932 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-host\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.630990 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631016 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632077 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631044 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-var-lib-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631069 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-ovn\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631095 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631121 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-k8s-cni-cncf-io\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631145 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-sys-fs\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631220 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-sys-fs\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631247 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-env-overrides\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-var-lib-cni-bin\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-ovn\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-run-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-multus-daemon-config\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631364 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631372 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3973fd7-657b-4408-87a3-48d5d15b2e21-var-lib-openvswitch\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631398 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-host-run-k8s-cni-cncf-io\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631402 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-serviceca\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovnkube-script-lib\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.632766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.631486 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-cnibin\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.633608 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.632933 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3973fd7-657b-4408-87a3-48d5d15b2e21-ovn-node-metrics-cert\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.633608 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.633020 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95f1b7b2-68c2-4b1d-8939-d348b6b5daf0-agent-certs\") pod \"konnectivity-agent-bxxbs\" (UID: \"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0\") " pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.636080 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.636058 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 23 17:41:43.636080 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.636081 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 23 17:41:43.636249 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.636094 2575 projected.go:194] Error preparing data for projected volume kube-api-access-hswgh for pod openshift-network-diagnostics/network-check-target-28blj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:43.636249 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:43.636163 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh podName:7a793770-244c-4445-a796-b64408321a9c nodeName:}" failed. No retries permitted until 2026-04-23 17:41:44.136145725 +0000 UTC m=+3.155598742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hswgh" (UniqueName: "kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh") pod "network-check-target-28blj" (UID: "7a793770-244c-4445-a796-b64408321a9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:43.638474 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.638450 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgwj\" (UniqueName: \"kubernetes.io/projected/a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1-kube-api-access-rrgwj\") pod \"aws-ebs-csi-driver-node-5bkht\" (UID: \"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.638728 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.638710 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mtjn\" (UniqueName: \"kubernetes.io/projected/97c7e75c-b4d5-4eef-b1fd-a9102c87fccc-kube-api-access-4mtjn\") pod \"multus-lvm6f\" (UID: \"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc\") " pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.638813 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.638773 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48sgp\" (UniqueName: \"kubernetes.io/projected/f3973fd7-657b-4408-87a3-48d5d15b2e21-kube-api-access-48sgp\") pod \"ovnkube-node-p7h4b\" (UID: \"f3973fd7-657b-4408-87a3-48d5d15b2e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:43.638813 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.638789 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk925\" (UniqueName: \"kubernetes.io/projected/9a5b8fa2-c8ef-4328-b02c-5663e2246f76-kube-api-access-bk925\") pod \"node-ca-xlkpg\" (UID: \"9a5b8fa2-c8ef-4328-b02c-5663e2246f76\") " pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.726965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.726933 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" Apr 23 17:41:43.734840 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.734807 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-728bd" Apr 23 17:41:43.741525 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.741500 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7svv" Apr 23 17:41:43.748213 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.748189 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fswp" Apr 23 17:41:43.754809 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.754790 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:41:43.762370 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.762344 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlkpg" Apr 23 17:41:43.768995 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.768972 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lvm6f" Apr 23 17:41:43.776748 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.776728 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" Apr 23 17:41:43.782374 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:43.782356 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:41:44.033211 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.033174 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:44.033368 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:44.033284 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:44.033368 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:44.033342 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:41:45.033325434 +0000 UTC m=+4.052778428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:44.234623 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.234582 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:44.234769 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:44.234743 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 23 17:41:44.234769 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:44.234771 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 23 17:41:44.234920 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:44.234784 2575 projected.go:194] Error preparing data for projected volume kube-api-access-hswgh for pod openshift-network-diagnostics/network-check-target-28blj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:44.234920 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:44.234845 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh podName:7a793770-244c-4445-a796-b64408321a9c nodeName:}" failed. No retries permitted until 2026-04-23 17:41:45.23482614 +0000 UTC m=+4.254279150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hswgh" (UniqueName: "kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh") pod "network-check-target-28blj" (UID: "7a793770-244c-4445-a796-b64408321a9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:44.242653 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.242628 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb810179_e96c_4f26_a7fc_e3006013faed.slice/crio-0b35e526b6e6930fa8ad6efc4ff4ade9ea3d33bd067a991bbccba64f30e6772e WatchSource:0}: Error finding container 0b35e526b6e6930fa8ad6efc4ff4ade9ea3d33bd067a991bbccba64f30e6772e: Status 404 returned error can't find the container with id 0b35e526b6e6930fa8ad6efc4ff4ade9ea3d33bd067a991bbccba64f30e6772e Apr 23 17:41:44.243724 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.243702 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3973fd7_657b_4408_87a3_48d5d15b2e21.slice/crio-ad6e4672a86f11894c003fbdf48f07b22957de44b373cb50c9b02f0b457aec61 WatchSource:0}: Error finding container ad6e4672a86f11894c003fbdf48f07b22957de44b373cb50c9b02f0b457aec61: Status 404 returned error can't find the container with id ad6e4672a86f11894c003fbdf48f07b22957de44b373cb50c9b02f0b457aec61 Apr 23 17:41:44.244582 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.244545 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1870fd4e_15fe_4c77_a974_dd5e1934339a.slice/crio-e705ee70faaf44e49dde05bf88ea9d7f7c3e1b83949a7e13e8f9585f050449df WatchSource:0}: Error finding container e705ee70faaf44e49dde05bf88ea9d7f7c3e1b83949a7e13e8f9585f050449df: Status 404 returned error can't find the container with id e705ee70faaf44e49dde05bf88ea9d7f7c3e1b83949a7e13e8f9585f050449df Apr 23 17:41:44.248036 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.248014 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806976ba_94a9_41ca_b9b6_10181690a08a.slice/crio-641a8b087e9dc594d02565ed9a146e7d4cf71bc440872595574454ec8b25aa6a WatchSource:0}: Error finding container 641a8b087e9dc594d02565ed9a146e7d4cf71bc440872595574454ec8b25aa6a: Status 404 returned error can't find the container with id 641a8b087e9dc594d02565ed9a146e7d4cf71bc440872595574454ec8b25aa6a Apr 23 17:41:44.249605 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.249567 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c7e75c_b4d5_4eef_b1fd_a9102c87fccc.slice/crio-31d5cd05531c8830023d0545fa0221a10a068a90ddbd060c3410c2eb1c0d0628 WatchSource:0}: Error finding container 31d5cd05531c8830023d0545fa0221a10a068a90ddbd060c3410c2eb1c0d0628: Status 404 returned error can't find the container with id 31d5cd05531c8830023d0545fa0221a10a068a90ddbd060c3410c2eb1c0d0628 Apr 23 17:41:44.251877 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.250842 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9db399_293d_460a_8a3e_294b3c70c30e.slice/crio-2206f041c723a6897d46da32b77236741b66f13ed78d15e5ab5d0f1b62d1279f WatchSource:0}: Error finding container 2206f041c723a6897d46da32b77236741b66f13ed78d15e5ab5d0f1b62d1279f: Status 404 returned error can't find the container with id 2206f041c723a6897d46da32b77236741b66f13ed78d15e5ab5d0f1b62d1279f Apr 23 17:41:44.251877 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.251620 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20ca6c6_f757_4ca7_ae17_4c2f1774e1e1.slice/crio-b75c03afb00322ad35961d8839a202dade29c1fcfef6dfdcc404783125bbdae4 WatchSource:0}: Error finding container b75c03afb00322ad35961d8839a202dade29c1fcfef6dfdcc404783125bbdae4: Status 404 returned error can't find the container with id b75c03afb00322ad35961d8839a202dade29c1fcfef6dfdcc404783125bbdae4 Apr 23 17:41:44.252525 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.252496 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5b8fa2_c8ef_4328_b02c_5663e2246f76.slice/crio-fc1add20202a8820cc005759b9899aa4d4648ce14be2304352b82f033d09bf28 WatchSource:0}: Error finding container fc1add20202a8820cc005759b9899aa4d4648ce14be2304352b82f033d09bf28: Status 404 returned error can't find the container with id fc1add20202a8820cc005759b9899aa4d4648ce14be2304352b82f033d09bf28 Apr 23 17:41:44.253388 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:41:44.253245 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f1b7b2_68c2_4b1d_8939_d348b6b5daf0.slice/crio-7925fcc6c214bafda2fc8f0cff781601633cd555e4c24a659dca9e86baaca89a WatchSource:0}: Error finding container 7925fcc6c214bafda2fc8f0cff781601633cd555e4c24a659dca9e86baaca89a: Status 404 returned error can't find the container with id 7925fcc6c214bafda2fc8f0cff781601633cd555e4c24a659dca9e86baaca89a Apr 23 17:41:44.451019 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.450809 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-22 17:36:42 +0000 UTC" deadline="2027-11-10 18:11:28.349866113 +0000 UTC" Apr 23 17:41:44.451019 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.451018 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13584h29m43.898853776s" Apr 23 17:41:44.599225 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.599103 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lvm6f" event={"ID":"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc","Type":"ContainerStarted","Data":"31d5cd05531c8830023d0545fa0221a10a068a90ddbd060c3410c2eb1c0d0628"} Apr 23 17:41:44.601140 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.601070 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxxbs" event={"ID":"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0","Type":"ContainerStarted","Data":"7925fcc6c214bafda2fc8f0cff781601633cd555e4c24a659dca9e86baaca89a"} Apr 23 17:41:44.602294 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.602238 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlkpg" event={"ID":"9a5b8fa2-c8ef-4328-b02c-5663e2246f76","Type":"ContainerStarted","Data":"fc1add20202a8820cc005759b9899aa4d4648ce14be2304352b82f033d09bf28"} Apr 23 17:41:44.604423 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.604400 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" event={"ID":"1870fd4e-15fe-4c77-a974-dd5e1934339a","Type":"ContainerStarted","Data":"e705ee70faaf44e49dde05bf88ea9d7f7c3e1b83949a7e13e8f9585f050449df"} Apr 23 17:41:44.607276 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.607245 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-728bd" event={"ID":"bb810179-e96c-4f26-a7fc-e3006013faed","Type":"ContainerStarted","Data":"0b35e526b6e6930fa8ad6efc4ff4ade9ea3d33bd067a991bbccba64f30e6772e"} Apr 23 17:41:44.609580 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.609558 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" event={"ID":"b84d58d2271f831c3e7ac872fa85f304","Type":"ContainerStarted","Data":"62c99ba0580bd9907d4b0c145d3c35fc0114d80fa258b4ee9f536b105f2f1e44"} Apr 23 17:41:44.611213 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.611189 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" event={"ID":"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1","Type":"ContainerStarted","Data":"b75c03afb00322ad35961d8839a202dade29c1fcfef6dfdcc404783125bbdae4"} Apr 23 17:41:44.612302 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.612282 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerStarted","Data":"2206f041c723a6897d46da32b77236741b66f13ed78d15e5ab5d0f1b62d1279f"} Apr 23 17:41:44.613857 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.613834 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fswp" event={"ID":"806976ba-94a9-41ca-b9b6-10181690a08a","Type":"ContainerStarted","Data":"641a8b087e9dc594d02565ed9a146e7d4cf71bc440872595574454ec8b25aa6a"} Apr 23 17:41:44.615161 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.615136 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"ad6e4672a86f11894c003fbdf48f07b22957de44b373cb50c9b02f0b457aec61"} Apr 23 17:41:44.625112 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:44.625017 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal" podStartSLOduration=2.625002593 podStartE2EDuration="2.625002593s" podCreationTimestamp="2026-04-23 17:41:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:41:44.624383731 +0000 UTC m=+3.643836748" watchObservedRunningTime="2026-04-23 17:41:44.625002593 +0000 UTC m=+3.644455609" Apr 23 17:41:45.040167 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:45.040132 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:45.040322 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.040303 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:45.040384 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.040376 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:41:47.040354439 +0000 UTC m=+6.059807452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:45.241888 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:45.241853 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:45.242048 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.242022 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 23 17:41:45.242048 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.242046 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 23 17:41:45.242164 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.242059 2575 projected.go:194] Error preparing data for projected volume kube-api-access-hswgh for pod openshift-network-diagnostics/network-check-target-28blj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:45.242164 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.242118 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh podName:7a793770-244c-4445-a796-b64408321a9c nodeName:}" failed. No retries permitted until 2026-04-23 17:41:47.242097493 +0000 UTC m=+6.261550504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hswgh" (UniqueName: "kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh") pod "network-check-target-28blj" (UID: "7a793770-244c-4445-a796-b64408321a9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:45.590290 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:45.590211 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:45.590735 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.590341 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:45.590735 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:45.590225 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:45.590735 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:45.590459 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:45.638940 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:45.638902 2575 generic.go:358] "Generic (PLEG): container finished" podID="0eff2e2d04276d2cb5575ebf2b719f08" containerID="de53c1d0f2e4f0fa1f6f28c0182f45f76e491863cfd56c401c3ac68f6a816201" exitCode=0 Apr 23 17:41:45.639813 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:45.639787 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" event={"ID":"0eff2e2d04276d2cb5575ebf2b719f08","Type":"ContainerDied","Data":"de53c1d0f2e4f0fa1f6f28c0182f45f76e491863cfd56c401c3ac68f6a816201"} Apr 23 17:41:46.649905 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:46.649856 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" event={"ID":"0eff2e2d04276d2cb5575ebf2b719f08","Type":"ContainerStarted","Data":"b2738e5c4235b614b360334f7cec888097658f5b35cf4228db4795b40780817a"} Apr 23 17:41:47.057636 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:47.057579 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:47.057826 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.057719 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:47.057826 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.057806 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:41:51.057784222 +0000 UTC m=+10.077237219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:47.259743 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:47.259707 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:47.259945 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.259919 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 23 17:41:47.260008 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.259955 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 23 17:41:47.260008 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.259969 2575 projected.go:194] Error preparing data for projected volume kube-api-access-hswgh for pod openshift-network-diagnostics/network-check-target-28blj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:47.260114 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.260038 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh podName:7a793770-244c-4445-a796-b64408321a9c nodeName:}" failed. No retries permitted until 2026-04-23 17:41:51.260016805 +0000 UTC m=+10.279469820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hswgh" (UniqueName: "kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh") pod "network-check-target-28blj" (UID: "7a793770-244c-4445-a796-b64408321a9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:47.591204 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:47.590684 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:47.591204 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.590838 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:47.591204 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:47.590907 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:47.591204 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:47.591017 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:49.591400 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:49.591252 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:49.591400 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:49.591283 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:49.591937 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:49.591405 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:49.591937 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:49.591841 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:51.091421 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:51.091382 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:51.091952 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.091537 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:51.091952 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.091612 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:41:59.091577246 +0000 UTC m=+18.111030261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:51.293146 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:51.293008 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:51.293347 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.293212 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 23 17:41:51.293347 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.293231 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 23 17:41:51.293347 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.293242 2575 projected.go:194] Error preparing data for projected volume kube-api-access-hswgh for pod openshift-network-diagnostics/network-check-target-28blj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:51.293347 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.293302 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh podName:7a793770-244c-4445-a796-b64408321a9c nodeName:}" failed. No retries permitted until 2026-04-23 17:41:59.293281694 +0000 UTC m=+18.312734688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hswgh" (UniqueName: "kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh") pod "network-check-target-28blj" (UID: "7a793770-244c-4445-a796-b64408321a9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:51.591642 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:51.590741 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:51.591642 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.590884 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:51.592086 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:51.592017 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:51.592086 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:51.592076 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:53.590880 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:53.590794 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:53.590880 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:53.590826 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:53.591315 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:53.590947 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:53.591315 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:53.591084 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:55.589907 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:55.589876 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:55.590448 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:55.589878 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:55.590448 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:55.590000 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:55.590448 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:55.590135 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:57.590870 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:57.590838 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:57.591294 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:57.590838 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:57.591294 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:57.590962 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:57.591294 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:57.591066 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:41:59.154337 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:59.154300 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:59.154806 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.154421 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:59.154806 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.154477 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:42:15.154459859 +0000 UTC m=+34.173912856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 23 17:41:59.355449 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:59.355406 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:59.355633 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.355583 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 23 17:41:59.355633 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.355621 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 23 17:41:59.355762 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.355636 2575 projected.go:194] Error preparing data for projected volume kube-api-access-hswgh for pod openshift-network-diagnostics/network-check-target-28blj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:59.355762 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.355698 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh podName:7a793770-244c-4445-a796-b64408321a9c nodeName:}" failed. No retries permitted until 2026-04-23 17:42:15.355679336 +0000 UTC m=+34.375132331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hswgh" (UniqueName: "kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh") pod "network-check-target-28blj" (UID: "7a793770-244c-4445-a796-b64408321a9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 23 17:41:59.590547 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:59.590512 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:41:59.590745 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:41:59.590512 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:41:59.590745 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.590683 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:41:59.590745 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:41:59.590723 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:01.590650 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.590464 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:01.591094 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.590526 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:01.591094 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:01.590778 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:01.591094 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:01.590827 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:01.676905 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.676878 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" event={"ID":"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1","Type":"ContainerStarted","Data":"fbbd4d6f024fa858e572a4a8feabcbc42d0b3ddb40a2ffded31f12ba1f9336a4"} Apr 23 17:42:01.677943 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.677921 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerStarted","Data":"af001aa47a9deadaf2c162859cff95fcff4dd95df28d8d2eaee29f89a1e51e28"} Apr 23 17:42:01.679448 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.679368 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:42:01.679708 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.679686 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"a78a12f6eabc9e040c637745f956a8c42c42b1d3161a311c840a79a40be84f23"} Apr 23 17:42:01.679783 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.679717 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerDied","Data":"261c82f986ecf4691a8852dd56827e18a475ccb69133b39ea7ebfc32ec15c466"} Apr 23 17:42:01.679783 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.679691 2575 generic.go:358] "Generic (PLEG): container finished" podID="f3973fd7-657b-4408-87a3-48d5d15b2e21" containerID="261c82f986ecf4691a8852dd56827e18a475ccb69133b39ea7ebfc32ec15c466" exitCode=1 Apr 23 17:42:01.679783 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.679767 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"1c877189ae00465cb2014dad93de512b2dc1fccb6c6c71e6c92c939d3e01549e"} Apr 23 17:42:01.680934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.680912 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lvm6f" event={"ID":"97c7e75c-b4d5-4eef-b1fd-a9102c87fccc","Type":"ContainerStarted","Data":"e34bab361b9c73d7edc6d0b6e87c4a26409592c9108ba345dd4ba1ff22b64295"} Apr 23 17:42:01.682152 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.682125 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxxbs" event={"ID":"95f1b7b2-68c2-4b1d-8939-d348b6b5daf0","Type":"ContainerStarted","Data":"212294b1ef48d1ccee15c997067aaf627ac9266da3868a72e46c0c82b68c08f5"} Apr 23 17:42:01.683385 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.683363 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlkpg" event={"ID":"9a5b8fa2-c8ef-4328-b02c-5663e2246f76","Type":"ContainerStarted","Data":"01bb798819497cbb5a132292e406905cc94c6efa09783bf4ad37b39728ab90cb"} Apr 23 17:42:01.684559 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.684536 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" event={"ID":"1870fd4e-15fe-4c77-a974-dd5e1934339a","Type":"ContainerStarted","Data":"f0b12ecc1cb78d0813b59b0dfee37c1cafd86598656ad5821fbb2e954cea5876"} Apr 23 17:42:01.687532 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.687512 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-728bd" event={"ID":"bb810179-e96c-4f26-a7fc-e3006013faed","Type":"ContainerStarted","Data":"255c02db263e4aa36fad1af56deea12e434ed54002d6ba9469a2e70c051f874d"} Apr 23 17:42:01.700343 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.700300 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-68.ec2.internal" podStartSLOduration=19.700289018 podStartE2EDuration="19.700289018s" podCreationTimestamp="2026-04-23 17:41:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:41:46.666893205 +0000 UTC m=+5.686346223" watchObservedRunningTime="2026-04-23 17:42:01.700289018 +0000 UTC m=+20.719742033" Apr 23 17:42:01.718763 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.718705 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rrrqf" podStartSLOduration=3.80558817 podStartE2EDuration="20.718688346s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.246759371 +0000 UTC m=+3.266212380" lastFinishedPulling="2026-04-23 17:42:01.159859547 +0000 UTC m=+20.179312556" observedRunningTime="2026-04-23 17:42:01.71852326 +0000 UTC m=+20.737976277" watchObservedRunningTime="2026-04-23 17:42:01.718688346 +0000 UTC m=+20.738141362" Apr 23 17:42:01.733873 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.733692 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-728bd" podStartSLOduration=4.004976792 podStartE2EDuration="20.733678427s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.245348017 +0000 UTC m=+3.264801011" lastFinishedPulling="2026-04-23 17:42:00.974049635 +0000 UTC m=+19.993502646" observedRunningTime="2026-04-23 17:42:01.733498882 +0000 UTC m=+20.752951898" watchObservedRunningTime="2026-04-23 17:42:01.733678427 +0000 UTC m=+20.753131443" Apr 23 17:42:01.748380 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.748329 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bxxbs" podStartSLOduration=8.378297304 podStartE2EDuration="20.74831271s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.276679518 +0000 UTC m=+3.296132512" lastFinishedPulling="2026-04-23 17:41:56.646694921 +0000 UTC m=+15.666147918" observedRunningTime="2026-04-23 17:42:01.747825202 +0000 UTC m=+20.767278220" watchObservedRunningTime="2026-04-23 17:42:01.74831271 +0000 UTC m=+20.767765738" Apr 23 17:42:01.765207 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.765166 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xlkpg" podStartSLOduration=4.067797276 podStartE2EDuration="20.765152383s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.276693611 +0000 UTC m=+3.296146612" lastFinishedPulling="2026-04-23 17:42:00.974048711 +0000 UTC m=+19.993501719" observedRunningTime="2026-04-23 17:42:01.764747899 +0000 UTC m=+20.784200916" watchObservedRunningTime="2026-04-23 17:42:01.765152383 +0000 UTC m=+20.784605410" Apr 23 17:42:01.782995 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:01.782955 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-lvm6f" podStartSLOduration=3.832607769 podStartE2EDuration="20.782938246s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.251778806 +0000 UTC m=+3.271231804" lastFinishedPulling="2026-04-23 17:42:01.202109273 +0000 UTC m=+20.221562281" observedRunningTime="2026-04-23 17:42:01.782420546 +0000 UTC m=+20.801873573" watchObservedRunningTime="2026-04-23 17:42:01.782938246 +0000 UTC m=+20.802391275" Apr 23 17:42:02.631881 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.631844 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 23 17:42:02.690348 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.690313 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" event={"ID":"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1","Type":"ContainerStarted","Data":"5ee361262cb099abc60743301f30fa4d23ad810c5709b21e4737dfd182991c18"} Apr 23 17:42:02.691666 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.691641 2575 generic.go:358] "Generic (PLEG): container finished" podID="7f9db399-293d-460a-8a3e-294b3c70c30e" containerID="af001aa47a9deadaf2c162859cff95fcff4dd95df28d8d2eaee29f89a1e51e28" exitCode=0 Apr 23 17:42:02.691754 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.691704 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerDied","Data":"af001aa47a9deadaf2c162859cff95fcff4dd95df28d8d2eaee29f89a1e51e28"} Apr 23 17:42:02.692965 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.692941 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fswp" event={"ID":"806976ba-94a9-41ca-b9b6-10181690a08a","Type":"ContainerStarted","Data":"8ffaa2e9dd5bd0a8b111c63c91aedaf4ad894f7a30e387b61816390a71e7b0c9"} Apr 23 17:42:02.695504 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.695486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:42:02.695944 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.695916 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"d4544f4072fa72cf9b5de1573ea58e3536cfd8b5e79728f694965ada32238532"} Apr 23 17:42:02.696004 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.695954 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"809ba5edf33aacc1658e7024b709e8dce7d14589281781485942464f690a2565"} Apr 23 17:42:02.696004 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.695967 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"135ea5e16e3544039adb4ec01fdf2734c158028ebc8ec04581bb37e76951ab7c"} Apr 23 17:42:02.726407 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:02.726351 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2fswp" podStartSLOduration=4.817103963 podStartE2EDuration="21.726338762s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.25060969 +0000 UTC m=+3.270062684" lastFinishedPulling="2026-04-23 17:42:01.159844472 +0000 UTC m=+20.179297483" observedRunningTime="2026-04-23 17:42:02.726179369 +0000 UTC m=+21.745632402" watchObservedRunningTime="2026-04-23 17:42:02.726338762 +0000 UTC m=+21.745791795" Apr 23 17:42:03.489009 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:03.488898 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-23T17:42:02.631872404Z","UUID":"4d0447da-3b70-4c16-81c6-5fcebf41cb73","Handler":null,"Name":"","Endpoint":""} Apr 23 17:42:03.491773 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:03.491751 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 23 17:42:03.491878 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:03.491780 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 23 17:42:03.589897 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:03.589858 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:03.590059 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:03.589984 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:03.590059 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:03.589993 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:03.590141 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:03.590087 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:04.703030 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:04.703006 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:42:04.703457 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:04.703386 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"c9f1e8329c10c8891c2b4075c4aa148333c0d91bc50643666de5bfd0359f7b8f"} Apr 23 17:42:04.705288 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:04.705259 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" event={"ID":"a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1","Type":"ContainerStarted","Data":"8a110a1c9944b585e92bfd6a9fcec6db0c08d30caf7aa6461fb386b54191b703"} Apr 23 17:42:04.775632 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:04.775574 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:42:04.776253 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:04.776230 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:42:04.793145 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:04.793102 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5bkht" podStartSLOduration=4.424562888 podStartE2EDuration="23.793088307s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.276720669 +0000 UTC m=+3.296173685" lastFinishedPulling="2026-04-23 17:42:03.645246102 +0000 UTC m=+22.664699104" observedRunningTime="2026-04-23 17:42:04.724304852 +0000 UTC m=+23.743757869" watchObservedRunningTime="2026-04-23 17:42:04.793088307 +0000 UTC m=+23.812541325" Apr 23 17:42:05.590356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:05.590117 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:05.590512 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:05.590124 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:05.590512 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:05.590455 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:05.590652 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:05.590524 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:05.707860 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:05.707829 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:42:05.708280 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:05.708259 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bxxbs" Apr 23 17:42:07.592547 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.592380 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:07.593270 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.592386 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:07.593270 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:07.592666 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:07.593270 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:07.592745 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:07.713173 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.713130 2575 generic.go:358] "Generic (PLEG): container finished" podID="7f9db399-293d-460a-8a3e-294b3c70c30e" containerID="f0af87013eb00986b9c2e01a3d32da0bcb71bdc5b0535621747f399d0593a26b" exitCode=0 Apr 23 17:42:07.713333 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.713218 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerDied","Data":"f0af87013eb00986b9c2e01a3d32da0bcb71bdc5b0535621747f399d0593a26b"} Apr 23 17:42:07.720743 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.720724 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:42:07.721276 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.721254 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"3e312c4557f2e39acc5d0c10b521c6d1351363140c04e116e09472cf6f680094"} Apr 23 17:42:07.721757 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.721719 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:42:07.721757 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.721746 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:42:07.721757 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.721758 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:42:07.721955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.721904 2575 scope.go:117] "RemoveContainer" containerID="261c82f986ecf4691a8852dd56827e18a475ccb69133b39ea7ebfc32ec15c466" Apr 23 17:42:07.736912 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.736887 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:42:07.737121 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:07.737110 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:42:08.725133 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.725105 2575 generic.go:358] "Generic (PLEG): container finished" podID="7f9db399-293d-460a-8a3e-294b3c70c30e" containerID="c33719f9666dc13038fbac29713872209a34d9774385af97aab53b0d68c9ab8f" exitCode=0 Apr 23 17:42:08.725627 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.725177 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerDied","Data":"c33719f9666dc13038fbac29713872209a34d9774385af97aab53b0d68c9ab8f"} Apr 23 17:42:08.728380 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.728364 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:42:08.728691 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.728666 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" event={"ID":"f3973fd7-657b-4408-87a3-48d5d15b2e21","Type":"ContainerStarted","Data":"b16a64484ac4fe47b909d3f003a325806d63f138aae69fdf35bd7c2b1f979ac0"} Apr 23 17:42:08.778406 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.778349 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" podStartSLOduration=10.795804175 podStartE2EDuration="27.778331044s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.24704975 +0000 UTC m=+3.266502758" lastFinishedPulling="2026-04-23 17:42:01.229576616 +0000 UTC m=+20.249029627" observedRunningTime="2026-04-23 17:42:08.777822295 +0000 UTC m=+27.797275315" watchObservedRunningTime="2026-04-23 17:42:08.778331044 +0000 UTC m=+27.797784061" Apr 23 17:42:08.791188 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.791158 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-28blj"] Apr 23 17:42:08.791368 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.791292 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:08.791432 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:08.791396 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:08.797966 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.797940 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-76492"] Apr 23 17:42:08.798092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:08.798075 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:08.798199 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:08.798179 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:09.732091 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:09.732060 2575 generic.go:358] "Generic (PLEG): container finished" podID="7f9db399-293d-460a-8a3e-294b3c70c30e" containerID="c86da88607fb52c01dda3c5f2e292712daa8ecac33aa390c7fb5defb39131fcf" exitCode=0 Apr 23 17:42:09.732469 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:09.732169 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerDied","Data":"c86da88607fb52c01dda3c5f2e292712daa8ecac33aa390c7fb5defb39131fcf"} Apr 23 17:42:10.590868 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:10.590837 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:10.591031 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:10.590953 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:10.591031 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:10.591004 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:10.591129 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:10.591098 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:12.590363 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:12.590144 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:12.590854 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:12.590209 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:12.590854 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:12.590451 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-28blj" podUID="7a793770-244c-4445-a796-b64408321a9c" Apr 23 17:42:12.590854 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:12.590552 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:42:14.257419 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.257390 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-68.ec2.internal" event="NodeReady" Apr 23 17:42:14.257986 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.257534 2575 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 23 17:42:14.307090 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.307005 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9786k"] Apr 23 17:42:14.311986 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.311961 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.314784 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.314758 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 23 17:42:14.314891 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.314857 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zw6qq\"" Apr 23 17:42:14.315144 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.315118 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 23 17:42:14.317289 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.317269 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bwrpd"] Apr 23 17:42:14.320841 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.320815 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:14.320969 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.320870 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9786k"] Apr 23 17:42:14.324236 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.324215 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 23 17:42:14.324329 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.324283 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-576jd\"" Apr 23 17:42:14.324329 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.324284 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 23 17:42:14.324509 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.324469 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 23 17:42:14.331501 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.331481 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bwrpd"] Apr 23 17:42:14.462457 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.462417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3459e220-189c-4bb2-91fd-ebd60ad6899e-config-volume\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.462648 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.462476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87dgr\" (UniqueName: \"kubernetes.io/projected/3459e220-189c-4bb2-91fd-ebd60ad6899e-kube-api-access-87dgr\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.462648 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.462553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:14.462648 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.462630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3459e220-189c-4bb2-91fd-ebd60ad6899e-tmp-dir\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.462764 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.462661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.462764 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.462686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdbm\" (UniqueName: \"kubernetes.io/projected/d2b26376-6f53-4edd-89f2-a4fe64523424-kube-api-access-jqdbm\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:14.563601 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.563504 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3459e220-189c-4bb2-91fd-ebd60ad6899e-config-volume\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.563601 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.563552 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-87dgr\" (UniqueName: \"kubernetes.io/projected/3459e220-189c-4bb2-91fd-ebd60ad6899e-kube-api-access-87dgr\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.563601 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.563576 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:14.563684 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.563693 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3459e220-189c-4bb2-91fd-ebd60ad6899e-tmp-dir\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.563729 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:14.563771 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:42:15.063751897 +0000 UTC m=+34.083204915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.563790 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdbm\" (UniqueName: \"kubernetes.io/projected/d2b26376-6f53-4edd-89f2-a4fe64523424-kube-api-access-jqdbm\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:14.563819 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:14.563860 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:14.563865 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:42:15.063854079 +0000 UTC m=+34.083307090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:14.564217 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.564003 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3459e220-189c-4bb2-91fd-ebd60ad6899e-tmp-dir\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.564217 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.564177 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3459e220-189c-4bb2-91fd-ebd60ad6899e-config-volume\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.576578 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.576545 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87dgr\" (UniqueName: \"kubernetes.io/projected/3459e220-189c-4bb2-91fd-ebd60ad6899e-kube-api-access-87dgr\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:14.576765 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.576620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdbm\" (UniqueName: \"kubernetes.io/projected/d2b26376-6f53-4edd-89f2-a4fe64523424-kube-api-access-jqdbm\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:14.590299 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.590270 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:14.590454 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.590303 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:14.592982 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.592961 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 23 17:42:14.593101 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.593063 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 23 17:42:14.593171 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.593066 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wdbsc\"" Apr 23 17:42:14.593235 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.593215 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 23 17:42:14.593235 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:14.593224 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gwf2x\"" Apr 23 17:42:15.066746 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.066706 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:15.066947 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.066760 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:15.066947 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:15.066886 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:15.066947 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:15.066900 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:15.067111 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:15.066970 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:42:16.066945353 +0000 UTC m=+35.086398360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:15.067111 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:15.066993 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:42:16.066976797 +0000 UTC m=+35.086429794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:15.167454 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.167415 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:15.167671 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:15.167604 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 23 17:42:15.167744 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:15.167684 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:42:47.167664282 +0000 UTC m=+66.187117276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : secret "metrics-daemon-secret" not found Apr 23 17:42:15.369774 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.369691 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:15.372376 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.372346 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hswgh\" (UniqueName: \"kubernetes.io/projected/7a793770-244c-4445-a796-b64408321a9c-kube-api-access-hswgh\") pod \"network-check-target-28blj\" (UID: \"7a793770-244c-4445-a796-b64408321a9c\") " pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:15.502428 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.502391 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:15.743402 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:15.743364 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-28blj"] Apr 23 17:42:15.747185 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:42:15.747159 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a793770_244c_4445_a796_b64408321a9c.slice/crio-4b00e4a82c6c8e58e6af5ad14be574885a4baff9de1f6394012abf94072d79b7 WatchSource:0}: Error finding container 4b00e4a82c6c8e58e6af5ad14be574885a4baff9de1f6394012abf94072d79b7: Status 404 returned error can't find the container with id 4b00e4a82c6c8e58e6af5ad14be574885a4baff9de1f6394012abf94072d79b7 Apr 23 17:42:16.074021 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:16.073988 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:16.074021 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:16.074026 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:16.074277 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:16.074156 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:16.074277 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:16.074216 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:16.074277 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:16.074222 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:42:18.074207586 +0000 UTC m=+37.093660581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:16.074399 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:16.074302 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:42:18.074271016 +0000 UTC m=+37.093724010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:16.749537 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:16.749496 2575 generic.go:358] "Generic (PLEG): container finished" podID="7f9db399-293d-460a-8a3e-294b3c70c30e" containerID="adf04378fb323b97376785300f1212deafe1cd8a9e77a8e5db1f9a6db072557c" exitCode=0 Apr 23 17:42:16.750135 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:16.749615 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerDied","Data":"adf04378fb323b97376785300f1212deafe1cd8a9e77a8e5db1f9a6db072557c"} Apr 23 17:42:16.750968 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:16.750734 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-28blj" event={"ID":"7a793770-244c-4445-a796-b64408321a9c","Type":"ContainerStarted","Data":"4b00e4a82c6c8e58e6af5ad14be574885a4baff9de1f6394012abf94072d79b7"} Apr 23 17:42:17.756241 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:17.756199 2575 generic.go:358] "Generic (PLEG): container finished" podID="7f9db399-293d-460a-8a3e-294b3c70c30e" containerID="058efef98674af7517561fd63e06496c67bdb7bbb3d54a02f5195195632911e4" exitCode=0 Apr 23 17:42:17.756241 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:17.756239 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerDied","Data":"058efef98674af7517561fd63e06496c67bdb7bbb3d54a02f5195195632911e4"} Apr 23 17:42:18.091769 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:18.091674 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:18.091769 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:18.091738 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:18.091987 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:18.091842 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:18.091987 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:18.091893 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:18.091987 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:18.091917 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:42:22.091896304 +0000 UTC m=+41.111349298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:18.091987 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:18.091956 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:42:22.091937892 +0000 UTC m=+41.111390886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:18.761940 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:18.761909 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7svv" event={"ID":"7f9db399-293d-460a-8a3e-294b3c70c30e","Type":"ContainerStarted","Data":"193c3e6683650abc64186d71987aa1660ec757c7e60d38efdbf74f119899cf36"} Apr 23 17:42:18.793014 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:18.792965 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7svv" podStartSLOduration=6.476960219 podStartE2EDuration="37.792950816s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:41:44.276735559 +0000 UTC m=+3.296188559" lastFinishedPulling="2026-04-23 17:42:15.592726149 +0000 UTC m=+34.612179156" observedRunningTime="2026-04-23 17:42:18.791668769 +0000 UTC m=+37.811121785" watchObservedRunningTime="2026-04-23 17:42:18.792950816 +0000 UTC m=+37.812403832" Apr 23 17:42:19.765325 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:19.765123 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-28blj" event={"ID":"7a793770-244c-4445-a796-b64408321a9c","Type":"ContainerStarted","Data":"2430cfca9d01286aac765f54e0754073a688f857fee239c10950f4a3e8ed949e"} Apr 23 17:42:19.765717 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:19.765605 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:42:19.781457 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:19.781405 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-28blj" podStartSLOduration=35.808905937 podStartE2EDuration="38.781390843s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:42:15.749058425 +0000 UTC m=+34.768511423" lastFinishedPulling="2026-04-23 17:42:18.721543325 +0000 UTC m=+37.740996329" observedRunningTime="2026-04-23 17:42:19.781266046 +0000 UTC m=+38.800719062" watchObservedRunningTime="2026-04-23 17:42:19.781390843 +0000 UTC m=+38.800843872" Apr 23 17:42:22.121277 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:22.121245 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:22.121651 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:22.121303 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:22.121651 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:22.121391 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:22.121651 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:22.121441 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:42:30.121427192 +0000 UTC m=+49.140880186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:22.121651 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:22.121392 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:22.121651 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:22.121525 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:42:30.121507883 +0000 UTC m=+49.140960883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:30.180425 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:30.180382 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:30.180425 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:30.180429 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:30.181000 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:30.180544 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:30.181000 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:30.180544 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:30.181000 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:30.180639 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:42:46.180618055 +0000 UTC m=+65.200071052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:30.181000 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:30.180676 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:42:46.180665024 +0000 UTC m=+65.200118032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:39.742760 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:39.742731 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7h4b" Apr 23 17:42:46.188738 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:46.188679 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:42:46.188738 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:46.188745 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:42:46.189229 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:46.188831 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:42:46.189229 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:46.188891 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:18.188876043 +0000 UTC m=+97.208329038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:42:46.189229 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:46.188840 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:42:46.189229 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:46.188964 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:43:18.188953357 +0000 UTC m=+97.208406351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:42:47.195927 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:47.195886 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:42:47.196303 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:47.195996 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 23 17:42:47.196303 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:42:47.196047 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:43:51.196032695 +0000 UTC m=+130.215485689 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : secret "metrics-daemon-secret" not found Apr 23 17:42:51.770795 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:42:51.770764 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-28blj" Apr 23 17:43:18.202442 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:18.202293 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:43:18.202442 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:18.202336 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:43:18.202442 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:18.202442 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 23 17:43:18.203016 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:18.202460 2575 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 23 17:43:18.203016 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:18.202505 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls podName:3459e220-189c-4bb2-91fd-ebd60ad6899e nodeName:}" failed. No retries permitted until 2026-04-23 17:44:22.202490599 +0000 UTC m=+161.221943593 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls") pod "dns-default-9786k" (UID: "3459e220-189c-4bb2-91fd-ebd60ad6899e") : secret "dns-default-metrics-tls" not found Apr 23 17:43:18.203016 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:18.202533 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert podName:d2b26376-6f53-4edd-89f2-a4fe64523424 nodeName:}" failed. No retries permitted until 2026-04-23 17:44:22.20251512 +0000 UTC m=+161.221968123 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert") pod "ingress-canary-bwrpd" (UID: "d2b26376-6f53-4edd-89f2-a4fe64523424") : secret "canary-serving-cert" not found Apr 23 17:43:27.508769 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.508733 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hhn8r"] Apr 23 17:43:27.511597 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.511561 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.514087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.514061 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 23 17:43:27.523864 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.523840 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hhn8r"] Apr 23 17:43:27.565210 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.565166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47d84ec3-de06-48f4-bafd-89def58eb514-original-pull-secret\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.565395 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.565250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47d84ec3-de06-48f4-bafd-89def58eb514-kubelet-config\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.565395 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.565274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47d84ec3-de06-48f4-bafd-89def58eb514-dbus\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.665800 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.665760 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47d84ec3-de06-48f4-bafd-89def58eb514-kubelet-config\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.665800 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.665800 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47d84ec3-de06-48f4-bafd-89def58eb514-dbus\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.666031 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.665861 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47d84ec3-de06-48f4-bafd-89def58eb514-original-pull-secret\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.666031 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.665871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47d84ec3-de06-48f4-bafd-89def58eb514-kubelet-config\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.666031 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.665991 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47d84ec3-de06-48f4-bafd-89def58eb514-dbus\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.669738 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.669713 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47d84ec3-de06-48f4-bafd-89def58eb514-original-pull-secret\") pod \"global-pull-secret-syncer-hhn8r\" (UID: \"47d84ec3-de06-48f4-bafd-89def58eb514\") " pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.820054 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.819951 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hhn8r" Apr 23 17:43:27.955233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:27.955203 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hhn8r"] Apr 23 17:43:27.958372 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:43:27.958340 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d84ec3_de06_48f4_bafd_89def58eb514.slice/crio-a7cf8f1c8e8f8334b6a8b29a77b7367b1685b06811e7ce2d08e4b1dd53e6a785 WatchSource:0}: Error finding container a7cf8f1c8e8f8334b6a8b29a77b7367b1685b06811e7ce2d08e4b1dd53e6a785: Status 404 returned error can't find the container with id a7cf8f1c8e8f8334b6a8b29a77b7367b1685b06811e7ce2d08e4b1dd53e6a785 Apr 23 17:43:28.900771 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:28.900729 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hhn8r" event={"ID":"47d84ec3-de06-48f4-bafd-89def58eb514","Type":"ContainerStarted","Data":"a7cf8f1c8e8f8334b6a8b29a77b7367b1685b06811e7ce2d08e4b1dd53e6a785"} Apr 23 17:43:31.908051 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:31.908016 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hhn8r" event={"ID":"47d84ec3-de06-48f4-bafd-89def58eb514","Type":"ContainerStarted","Data":"fa5c5f4e8b2ab2918376b43cd9b1362a3d95c8518ccb000c20e964ad816bd3aa"} Apr 23 17:43:31.928435 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:31.927102 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hhn8r" podStartSLOduration=1.507017426 podStartE2EDuration="4.927084317s" podCreationTimestamp="2026-04-23 17:43:27 +0000 UTC" firstStartedPulling="2026-04-23 17:43:27.959889324 +0000 UTC m=+106.979342318" lastFinishedPulling="2026-04-23 17:43:31.379956212 +0000 UTC m=+110.399409209" observedRunningTime="2026-04-23 17:43:31.92605399 +0000 UTC m=+110.945507027" watchObservedRunningTime="2026-04-23 17:43:31.927084317 +0000 UTC m=+110.946537333" Apr 23 17:43:40.491067 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.491031 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg"] Apr 23 17:43:40.493787 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.493768 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" Apr 23 17:43:40.497638 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.497618 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tcc4k\"" Apr 23 17:43:40.498227 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.498212 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Apr 23 17:43:40.498293 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.498217 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Apr 23 17:43:40.505070 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.505051 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg"] Apr 23 17:43:40.551942 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.551914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk826\" (UniqueName: \"kubernetes.io/projected/5c365186-3cfd-4cd1-868f-72da2be8292b-kube-api-access-sk826\") pod \"volume-data-source-validator-7c6cbb6c87-qgzxg\" (UID: \"5c365186-3cfd-4cd1-868f-72da2be8292b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" Apr 23 17:43:40.591280 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.591254 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8"] Apr 23 17:43:40.594080 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.594059 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.594941 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.594924 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-747f569c94-mjbpd"] Apr 23 17:43:40.597359 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.597336 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.599865 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.599845 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Apr 23 17:43:40.600010 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.599868 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 23 17:43:40.600103 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.599871 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Apr 23 17:43:40.600103 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.599893 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 23 17:43:40.600748 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.600726 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qdc79\"" Apr 23 17:43:40.600971 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.600958 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Apr 23 17:43:40.601053 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.601038 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wcqss\"" Apr 23 17:43:40.601106 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.601054 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Apr 23 17:43:40.601106 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.601055 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 23 17:43:40.601180 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.601059 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Apr 23 17:43:40.602649 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.602633 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 23 17:43:40.605693 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.605671 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Apr 23 17:43:40.618646 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.618623 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8"] Apr 23 17:43:40.619266 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.619248 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-747f569c94-mjbpd"] Apr 23 17:43:40.653117 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qp6\" (UniqueName: \"kubernetes.io/projected/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-kube-api-access-v2qp6\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.653117 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.653326 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-default-certificate\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.653326 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkgm\" (UniqueName: \"kubernetes.io/projected/eb0a83a2-e27a-459f-a491-f2ffb4073120-kube-api-access-5hkgm\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.653326 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-stats-auth\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.653326 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653302 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sk826\" (UniqueName: \"kubernetes.io/projected/5c365186-3cfd-4cd1-868f-72da2be8292b-kube-api-access-sk826\") pod \"volume-data-source-validator-7c6cbb6c87-qgzxg\" (UID: \"5c365186-3cfd-4cd1-868f-72da2be8292b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" Apr 23 17:43:40.653326 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb0a83a2-e27a-459f-a491-f2ffb4073120-telemetry-config\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.653549 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.653549 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.653393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.687664 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.687631 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk826\" (UniqueName: \"kubernetes.io/projected/5c365186-3cfd-4cd1-868f-72da2be8292b-kube-api-access-sk826\") pod \"volume-data-source-validator-7c6cbb6c87-qgzxg\" (UID: \"5c365186-3cfd-4cd1-868f-72da2be8292b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" Apr 23 17:43:40.754223 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754133 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-default-certificate\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.754223 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754200 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkgm\" (UniqueName: \"kubernetes.io/projected/eb0a83a2-e27a-459f-a491-f2ffb4073120-kube-api-access-5hkgm\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.754223 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754219 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-stats-auth\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.754482 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754250 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb0a83a2-e27a-459f-a491-f2ffb4073120-telemetry-config\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.754482 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754271 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.754482 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754290 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.754482 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754321 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qp6\" (UniqueName: \"kubernetes.io/projected/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-kube-api-access-v2qp6\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.754482 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:40.754464 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:41.254440489 +0000 UTC m=+120.273893507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : configmap references non-existent config key: service-ca.crt Apr 23 17:43:40.754796 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754534 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.754796 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:40.754666 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 23 17:43:40.754796 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:40.754687 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:40.754796 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:40.754743 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:41.254728117 +0000 UTC m=+120.274181126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : secret "router-metrics-certs-default" not found Apr 23 17:43:40.754796 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:40.754759 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls podName:eb0a83a2-e27a-459f-a491-f2ffb4073120 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:41.254751701 +0000 UTC m=+120.274204698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-5cln8" (UID: "eb0a83a2-e27a-459f-a491-f2ffb4073120") : secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:40.755006 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.754989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb0a83a2-e27a-459f-a491-f2ffb4073120-telemetry-config\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.756718 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.756692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-default-certificate\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.756826 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.756746 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-stats-auth\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.763715 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.763686 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkgm\" (UniqueName: \"kubernetes.io/projected/eb0a83a2-e27a-459f-a491-f2ffb4073120-kube-api-access-5hkgm\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:40.764011 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.763994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qp6\" (UniqueName: \"kubernetes.io/projected/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-kube-api-access-v2qp6\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:40.802377 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.802335 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" Apr 23 17:43:40.912547 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.912513 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg"] Apr 23 17:43:40.914998 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:43:40.914975 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c365186_3cfd_4cd1_868f_72da2be8292b.slice/crio-4462d588db5937c2e2056759d3b2e1c259dfe4fc2045f028a2dfc629198c0c4f WatchSource:0}: Error finding container 4462d588db5937c2e2056759d3b2e1c259dfe4fc2045f028a2dfc629198c0c4f: Status 404 returned error can't find the container with id 4462d588db5937c2e2056759d3b2e1c259dfe4fc2045f028a2dfc629198c0c4f Apr 23 17:43:40.926127 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:40.926101 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" event={"ID":"5c365186-3cfd-4cd1-868f-72da2be8292b","Type":"ContainerStarted","Data":"4462d588db5937c2e2056759d3b2e1c259dfe4fc2045f028a2dfc629198c0c4f"} Apr 23 17:43:41.259707 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:41.259678 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:41.259864 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:41.259713 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:41.259864 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:41.259738 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:41.259864 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:41.259818 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:41.259864 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:41.259820 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 23 17:43:41.259864 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:41.259866 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls podName:eb0a83a2-e27a-459f-a491-f2ffb4073120 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:42.25985317 +0000 UTC m=+121.279306164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-5cln8" (UID: "eb0a83a2-e27a-459f-a491-f2ffb4073120") : secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:41.260057 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:41.259879 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:42.259872651 +0000 UTC m=+121.279325644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : secret "router-metrics-certs-default" not found Apr 23 17:43:41.260057 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:41.259890 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:42.259885479 +0000 UTC m=+121.279338473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : configmap references non-existent config key: service-ca.crt Apr 23 17:43:42.267846 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:42.267807 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:42.267846 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:42.267851 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:42.268252 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:42.267887 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:42.268252 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:42.267970 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:42.268252 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:42.267971 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 23 17:43:42.268252 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:42.268024 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls podName:eb0a83a2-e27a-459f-a491-f2ffb4073120 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:44.268006146 +0000 UTC m=+123.287459159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-5cln8" (UID: "eb0a83a2-e27a-459f-a491-f2ffb4073120") : secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:42.268252 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:42.268047 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:44.268036959 +0000 UTC m=+123.287489955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : configmap references non-existent config key: service-ca.crt Apr 23 17:43:42.268252 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:42.268068 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:44.268061507 +0000 UTC m=+123.287514501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : secret "router-metrics-certs-default" not found Apr 23 17:43:42.931526 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:42.931486 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" event={"ID":"5c365186-3cfd-4cd1-868f-72da2be8292b","Type":"ContainerStarted","Data":"377e55e9ff7061ee16008906c1d5be90b717db2419dd30f1c7669a0bd9c7e2f5"} Apr 23 17:43:42.946804 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:42.946739 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-qgzxg" podStartSLOduration=1.835505882 podStartE2EDuration="2.946718907s" podCreationTimestamp="2026-04-23 17:43:40 +0000 UTC" firstStartedPulling="2026-04-23 17:43:40.916737258 +0000 UTC m=+119.936190251" lastFinishedPulling="2026-04-23 17:43:42.027950282 +0000 UTC m=+121.047403276" observedRunningTime="2026-04-23 17:43:42.946697078 +0000 UTC m=+121.966150108" watchObservedRunningTime="2026-04-23 17:43:42.946718907 +0000 UTC m=+121.966171925" Apr 23 17:43:44.284554 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.284514 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:44.284554 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.284558 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:44.284982 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.284613 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:44.284982 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:44.284678 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 23 17:43:44.284982 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:44.284699 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:44.284982 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:44.284750 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:48.284734379 +0000 UTC m=+127.304187377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : secret "router-metrics-certs-default" not found Apr 23 17:43:44.284982 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:44.284763 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:48.284757612 +0000 UTC m=+127.304210605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : configmap references non-existent config key: service-ca.crt Apr 23 17:43:44.284982 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:44.284773 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls podName:eb0a83a2-e27a-459f-a491-f2ffb4073120 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:48.284768662 +0000 UTC m=+127.304221656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-5cln8" (UID: "eb0a83a2-e27a-459f-a491-f2ffb4073120") : secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:44.365677 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.365648 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp"] Apr 23 17:43:44.368279 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.368265 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" Apr 23 17:43:44.370281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.370263 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zbqxs\"" Apr 23 17:43:44.381269 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.381247 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp"] Apr 23 17:43:44.486779 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.486736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7bq\" (UniqueName: \"kubernetes.io/projected/7df751b0-66d5-41a6-96da-5321b6c5c0ea-kube-api-access-hn7bq\") pod \"network-check-source-8894fc9bd-rltbp\" (UID: \"7df751b0-66d5-41a6-96da-5321b6c5c0ea\") " pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" Apr 23 17:43:44.587673 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.587562 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7bq\" (UniqueName: \"kubernetes.io/projected/7df751b0-66d5-41a6-96da-5321b6c5c0ea-kube-api-access-hn7bq\") pod \"network-check-source-8894fc9bd-rltbp\" (UID: \"7df751b0-66d5-41a6-96da-5321b6c5c0ea\") " pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" Apr 23 17:43:44.596437 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.596412 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7bq\" (UniqueName: \"kubernetes.io/projected/7df751b0-66d5-41a6-96da-5321b6c5c0ea-kube-api-access-hn7bq\") pod \"network-check-source-8894fc9bd-rltbp\" (UID: \"7df751b0-66d5-41a6-96da-5321b6c5c0ea\") " pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" Apr 23 17:43:44.676742 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.676699 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" Apr 23 17:43:44.787985 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.786557 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp"] Apr 23 17:43:44.790656 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:43:44.790624 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df751b0_66d5_41a6_96da_5321b6c5c0ea.slice/crio-e88a749ad213817370afa97196ea557ed6b33fdf0a68d44379ba661f44bd56bf WatchSource:0}: Error finding container e88a749ad213817370afa97196ea557ed6b33fdf0a68d44379ba661f44bd56bf: Status 404 returned error can't find the container with id e88a749ad213817370afa97196ea557ed6b33fdf0a68d44379ba661f44bd56bf Apr 23 17:43:44.936562 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.936526 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" event={"ID":"7df751b0-66d5-41a6-96da-5321b6c5c0ea","Type":"ContainerStarted","Data":"8f6e2154cf03a85f5a40e2cde8e5dec4ec10d9f657f8b3171573d320d762eeb4"} Apr 23 17:43:44.936562 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.936566 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" event={"ID":"7df751b0-66d5-41a6-96da-5321b6c5c0ea","Type":"ContainerStarted","Data":"e88a749ad213817370afa97196ea557ed6b33fdf0a68d44379ba661f44bd56bf"} Apr 23 17:43:44.955982 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:44.955936 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-8894fc9bd-rltbp" podStartSLOduration=0.955921901 podStartE2EDuration="955.921901ms" podCreationTimestamp="2026-04-23 17:43:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:43:44.955620993 +0000 UTC m=+123.975074009" watchObservedRunningTime="2026-04-23 17:43:44.955921901 +0000 UTC m=+123.975374919" Apr 23 17:43:46.324989 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.324961 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-728bd_bb810179-e96c-4f26-a7fc-e3006013faed/dns-node-resolver/0.log" Apr 23 17:43:46.408261 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.408231 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5"] Apr 23 17:43:46.411150 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.411135 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.413228 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.413208 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Apr 23 17:43:46.413228 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.413218 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Apr 23 17:43:46.413370 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.413340 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Apr 23 17:43:46.413413 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.413376 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Apr 23 17:43:46.413840 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.413820 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-f7bdp\"" Apr 23 17:43:46.421642 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.421622 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5"] Apr 23 17:43:46.502865 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.502823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e27305-61d1-45ca-a35f-ce823c1e6435-config\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.502865 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.502882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66e27305-61d1-45ca-a35f-ce823c1e6435-serving-cert\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.503074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.502945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzswn\" (UniqueName: \"kubernetes.io/projected/66e27305-61d1-45ca-a35f-ce823c1e6435-kube-api-access-lzswn\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.603459 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.603377 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66e27305-61d1-45ca-a35f-ce823c1e6435-serving-cert\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.603459 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.603415 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lzswn\" (UniqueName: \"kubernetes.io/projected/66e27305-61d1-45ca-a35f-ce823c1e6435-kube-api-access-lzswn\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.603653 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.603500 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e27305-61d1-45ca-a35f-ce823c1e6435-config\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.604117 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.604098 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e27305-61d1-45ca-a35f-ce823c1e6435-config\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.605684 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.605654 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66e27305-61d1-45ca-a35f-ce823c1e6435-serving-cert\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.611857 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.611837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzswn\" (UniqueName: \"kubernetes.io/projected/66e27305-61d1-45ca-a35f-ce823c1e6435-kube-api-access-lzswn\") pod \"service-ca-operator-d6fc45fc5-fc8h5\" (UID: \"66e27305-61d1-45ca-a35f-ce823c1e6435\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.719170 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.719134 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" Apr 23 17:43:46.832218 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.832188 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5"] Apr 23 17:43:46.835151 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:43:46.835115 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e27305_61d1_45ca_a35f_ce823c1e6435.slice/crio-0e04370f88c4e63dfc6e3a724c45175ab53f07a7f1d41a5894db11609babe403 WatchSource:0}: Error finding container 0e04370f88c4e63dfc6e3a724c45175ab53f07a7f1d41a5894db11609babe403: Status 404 returned error can't find the container with id 0e04370f88c4e63dfc6e3a724c45175ab53f07a7f1d41a5894db11609babe403 Apr 23 17:43:46.943218 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:46.943139 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" event={"ID":"66e27305-61d1-45ca-a35f-ce823c1e6435","Type":"ContainerStarted","Data":"0e04370f88c4e63dfc6e3a724c45175ab53f07a7f1d41a5894db11609babe403"} Apr 23 17:43:47.725439 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:47.725331 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xlkpg_9a5b8fa2-c8ef-4328-b02c-5663e2246f76/node-ca/0.log" Apr 23 17:43:48.318961 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.318933 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:48.319082 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.318990 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:48.319146 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.319092 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:48.319202 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:48.319140 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:56.319115904 +0000 UTC m=+135.338568915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : configmap references non-existent config key: service-ca.crt Apr 23 17:43:48.319202 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:48.319171 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:48.319202 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:48.319198 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 23 17:43:48.319356 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:48.319231 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls podName:eb0a83a2-e27a-459f-a491-f2ffb4073120 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:56.319213918 +0000 UTC m=+135.338667118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-5cln8" (UID: "eb0a83a2-e27a-459f-a491-f2ffb4073120") : secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:48.319356 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:48.319251 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:43:56.31924174 +0000 UTC m=+135.338694735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : secret "router-metrics-certs-default" not found Apr 23 17:43:48.431691 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.431656 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf"] Apr 23 17:43:48.434689 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.434668 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" Apr 23 17:43:48.436833 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.436810 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Apr 23 17:43:48.436941 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.436901 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Apr 23 17:43:48.440989 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.437623 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8tr5b\"" Apr 23 17:43:48.445068 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.445044 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf"] Apr 23 17:43:48.520727 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.520696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtb6\" (UniqueName: \"kubernetes.io/projected/83b95dec-6e53-4691-bce3-2f6d74fabb90-kube-api-access-kgtb6\") pod \"migrator-74bb7799d9-d6bzf\" (UID: \"83b95dec-6e53-4691-bce3-2f6d74fabb90\") " pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" Apr 23 17:43:48.621820 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.621742 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtb6\" (UniqueName: \"kubernetes.io/projected/83b95dec-6e53-4691-bce3-2f6d74fabb90-kube-api-access-kgtb6\") pod \"migrator-74bb7799d9-d6bzf\" (UID: \"83b95dec-6e53-4691-bce3-2f6d74fabb90\") " pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" Apr 23 17:43:48.629423 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.629392 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtb6\" (UniqueName: \"kubernetes.io/projected/83b95dec-6e53-4691-bce3-2f6d74fabb90-kube-api-access-kgtb6\") pod \"migrator-74bb7799d9-d6bzf\" (UID: \"83b95dec-6e53-4691-bce3-2f6d74fabb90\") " pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" Apr 23 17:43:48.725717 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.725684 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-7c6cbb6c87-qgzxg_5c365186-3cfd-4cd1-868f-72da2be8292b/volume-data-source-validator/0.log" Apr 23 17:43:48.747386 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.747356 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" Apr 23 17:43:48.864643 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.864522 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf"] Apr 23 17:43:48.866969 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:43:48.866941 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b95dec_6e53_4691_bce3_2f6d74fabb90.slice/crio-8db3f11a3d8648ee3e52c6abc2d188c4fd66baeb37f01932b2bb1afbaec1ce02 WatchSource:0}: Error finding container 8db3f11a3d8648ee3e52c6abc2d188c4fd66baeb37f01932b2bb1afbaec1ce02: Status 404 returned error can't find the container with id 8db3f11a3d8648ee3e52c6abc2d188c4fd66baeb37f01932b2bb1afbaec1ce02 Apr 23 17:43:48.925214 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.925150 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5bkht_a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1/csi-driver/0.log" Apr 23 17:43:48.948183 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.948147 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" event={"ID":"83b95dec-6e53-4691-bce3-2f6d74fabb90","Type":"ContainerStarted","Data":"8db3f11a3d8648ee3e52c6abc2d188c4fd66baeb37f01932b2bb1afbaec1ce02"} Apr 23 17:43:48.949405 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.949382 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" event={"ID":"66e27305-61d1-45ca-a35f-ce823c1e6435","Type":"ContainerStarted","Data":"5df342216a541a46898516aa0cd3e92db461bc28369f674e979b4b0f1fd96fd2"} Apr 23 17:43:48.966984 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:48.966943 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" podStartSLOduration=1.50220633 podStartE2EDuration="2.9669309s" podCreationTimestamp="2026-04-23 17:43:46 +0000 UTC" firstStartedPulling="2026-04-23 17:43:46.83687645 +0000 UTC m=+125.856329443" lastFinishedPulling="2026-04-23 17:43:48.301601012 +0000 UTC m=+127.321054013" observedRunningTime="2026-04-23 17:43:48.966096855 +0000 UTC m=+127.985549872" watchObservedRunningTime="2026-04-23 17:43:48.9669309 +0000 UTC m=+127.986383917" Apr 23 17:43:49.125096 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:49.125059 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5bkht_a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1/csi-node-driver-registrar/0.log" Apr 23 17:43:49.325189 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:49.325156 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5bkht_a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1/csi-liveness-probe/0.log" Apr 23 17:43:49.953415 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:49.953380 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" event={"ID":"83b95dec-6e53-4691-bce3-2f6d74fabb90","Type":"ContainerStarted","Data":"7aac4ee6346eeb81d24c882a181011f824170c9596861c86a522cf4a9596c0d0"} Apr 23 17:43:49.953415 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:49.953417 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" event={"ID":"83b95dec-6e53-4691-bce3-2f6d74fabb90","Type":"ContainerStarted","Data":"efc863d14e3c81a2dc6f43f7e4dd3d20c98d94cadf702a04a0a8fdfcf6e50625"} Apr 23 17:43:49.969846 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:49.969799 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-d6bzf" podStartSLOduration=1.070148424 podStartE2EDuration="1.969784577s" podCreationTimestamp="2026-04-23 17:43:48 +0000 UTC" firstStartedPulling="2026-04-23 17:43:48.868761704 +0000 UTC m=+127.888214698" lastFinishedPulling="2026-04-23 17:43:49.768397844 +0000 UTC m=+128.787850851" observedRunningTime="2026-04-23 17:43:49.96896211 +0000 UTC m=+128.988415133" watchObservedRunningTime="2026-04-23 17:43:49.969784577 +0000 UTC m=+128.989237592" Apr 23 17:43:51.243202 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:51.243169 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:43:51.243563 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:51.243282 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 23 17:43:51.243563 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:51.243337 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs podName:eb1ab24a-5f1e-4324-9b5f-f92abb7647fa nodeName:}" failed. No retries permitted until 2026-04-23 17:45:53.243324111 +0000 UTC m=+252.262777104 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs") pod "network-metrics-daemon-76492" (UID: "eb1ab24a-5f1e-4324-9b5f-f92abb7647fa") : secret "metrics-daemon-secret" not found Apr 23 17:43:52.309870 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.309837 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-865cb79987-x4ng7"] Apr 23 17:43:52.312924 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.312905 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.315741 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.315716 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Apr 23 17:43:52.315741 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.315729 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Apr 23 17:43:52.315894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.315772 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Apr 23 17:43:52.316040 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.316024 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gm8pr\"" Apr 23 17:43:52.316100 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.316087 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Apr 23 17:43:52.322216 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.322195 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-x4ng7"] Apr 23 17:43:52.451682 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.451645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcz4\" (UniqueName: \"kubernetes.io/projected/48965f97-aeed-42df-bab1-c53246a5bd45-kube-api-access-fqcz4\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.451859 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.451696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48965f97-aeed-42df-bab1-c53246a5bd45-signing-cabundle\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.451859 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.451783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48965f97-aeed-42df-bab1-c53246a5bd45-signing-key\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.552083 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.552049 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48965f97-aeed-42df-bab1-c53246a5bd45-signing-key\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.552278 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.552142 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcz4\" (UniqueName: \"kubernetes.io/projected/48965f97-aeed-42df-bab1-c53246a5bd45-kube-api-access-fqcz4\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.552278 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.552174 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48965f97-aeed-42df-bab1-c53246a5bd45-signing-cabundle\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.552908 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.552887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48965f97-aeed-42df-bab1-c53246a5bd45-signing-cabundle\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.554521 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.554496 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48965f97-aeed-42df-bab1-c53246a5bd45-signing-key\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.560992 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.560942 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcz4\" (UniqueName: \"kubernetes.io/projected/48965f97-aeed-42df-bab1-c53246a5bd45-kube-api-access-fqcz4\") pod \"service-ca-865cb79987-x4ng7\" (UID: \"48965f97-aeed-42df-bab1-c53246a5bd45\") " pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.622219 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.622190 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-x4ng7" Apr 23 17:43:52.735671 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.735643 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-x4ng7"] Apr 23 17:43:52.739161 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:43:52.739127 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48965f97_aeed_42df_bab1_c53246a5bd45.slice/crio-ed164b77fa86459f41a842269b98d7c43410207b56279d8ed54c29ee537bd377 WatchSource:0}: Error finding container ed164b77fa86459f41a842269b98d7c43410207b56279d8ed54c29ee537bd377: Status 404 returned error can't find the container with id ed164b77fa86459f41a842269b98d7c43410207b56279d8ed54c29ee537bd377 Apr 23 17:43:52.962062 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.961978 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-x4ng7" event={"ID":"48965f97-aeed-42df-bab1-c53246a5bd45","Type":"ContainerStarted","Data":"b179cec77cd489604d9785ac3ee979d7c57ef083beae417dc47976b21c1c079e"} Apr 23 17:43:52.962062 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.962022 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-x4ng7" event={"ID":"48965f97-aeed-42df-bab1-c53246a5bd45","Type":"ContainerStarted","Data":"ed164b77fa86459f41a842269b98d7c43410207b56279d8ed54c29ee537bd377"} Apr 23 17:43:52.979532 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:52.979486 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-865cb79987-x4ng7" podStartSLOduration=0.979469211 podStartE2EDuration="979.469211ms" podCreationTimestamp="2026-04-23 17:43:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:43:52.978701585 +0000 UTC m=+131.998154613" watchObservedRunningTime="2026-04-23 17:43:52.979469211 +0000 UTC m=+131.998922237" Apr 23 17:43:56.390503 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:56.390464 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:56.390503 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:56.390509 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:43:56.390986 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:43:56.390533 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:43:56.390986 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:56.390630 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 23 17:43:56.390986 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:56.390676 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:44:12.390657736 +0000 UTC m=+151.410110730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : configmap references non-existent config key: service-ca.crt Apr 23 17:43:56.390986 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:56.390700 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs podName:2dc66a6c-a16e-41b1-9fd0-cd602ca9e666 nodeName:}" failed. No retries permitted until 2026-04-23 17:44:12.39069232 +0000 UTC m=+151.410145314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs") pod "router-default-747f569c94-mjbpd" (UID: "2dc66a6c-a16e-41b1-9fd0-cd602ca9e666") : secret "router-metrics-certs-default" not found Apr 23 17:43:56.390986 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:56.390637 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 23 17:43:56.390986 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:43:56.390727 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls podName:eb0a83a2-e27a-459f-a491-f2ffb4073120 nodeName:}" failed. No retries permitted until 2026-04-23 17:44:12.390720975 +0000 UTC m=+151.410173968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-5cln8" (UID: "eb0a83a2-e27a-459f-a491-f2ffb4073120") : secret "cluster-monitoring-operator-tls" not found Apr 23 17:44:12.412842 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.412796 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:12.412842 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.412847 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:12.413375 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.412941 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:44:12.413564 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.413541 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-service-ca-bundle\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:12.415184 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.415165 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc66a6c-a16e-41b1-9fd0-cd602ca9e666-metrics-certs\") pod \"router-default-747f569c94-mjbpd\" (UID: \"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666\") " pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:12.415310 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.415294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb0a83a2-e27a-459f-a491-f2ffb4073120-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-5cln8\" (UID: \"eb0a83a2-e27a-459f-a491-f2ffb4073120\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:44:12.706733 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.706658 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wcqss\"" Apr 23 17:44:12.709496 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.709478 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qdc79\"" Apr 23 17:44:12.714096 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.714072 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" Apr 23 17:44:12.718750 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.718734 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:12.839548 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.839413 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8"] Apr 23 17:44:12.842276 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:44:12.842253 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a83a2_e27a_459f_a491_f2ffb4073120.slice/crio-72d4f836cfc5794e8614f94957cc996a7b2545738f1a19aea32f1fec4b0be008 WatchSource:0}: Error finding container 72d4f836cfc5794e8614f94957cc996a7b2545738f1a19aea32f1fec4b0be008: Status 404 returned error can't find the container with id 72d4f836cfc5794e8614f94957cc996a7b2545738f1a19aea32f1fec4b0be008 Apr 23 17:44:12.854971 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:12.854947 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-747f569c94-mjbpd"] Apr 23 17:44:12.857560 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:44:12.857534 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc66a6c_a16e_41b1_9fd0_cd602ca9e666.slice/crio-27c9ae78cb5539ed72d1a87c5b7dcaaf0e8354af8e32f70706cd3d9041f0044a WatchSource:0}: Error finding container 27c9ae78cb5539ed72d1a87c5b7dcaaf0e8354af8e32f70706cd3d9041f0044a: Status 404 returned error can't find the container with id 27c9ae78cb5539ed72d1a87c5b7dcaaf0e8354af8e32f70706cd3d9041f0044a Apr 23 17:44:13.013441 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:13.013402 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-747f569c94-mjbpd" event={"ID":"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666","Type":"ContainerStarted","Data":"57031251c8d590fe52e5ceb413d4fbeb35678cbbc3d4ce395fbeea4fc1dad3a3"} Apr 23 17:44:13.013652 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:13.013449 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-747f569c94-mjbpd" event={"ID":"2dc66a6c-a16e-41b1-9fd0-cd602ca9e666","Type":"ContainerStarted","Data":"27c9ae78cb5539ed72d1a87c5b7dcaaf0e8354af8e32f70706cd3d9041f0044a"} Apr 23 17:44:13.014474 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:13.014451 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" event={"ID":"eb0a83a2-e27a-459f-a491-f2ffb4073120","Type":"ContainerStarted","Data":"72d4f836cfc5794e8614f94957cc996a7b2545738f1a19aea32f1fec4b0be008"} Apr 23 17:44:13.035817 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:13.035777 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-747f569c94-mjbpd" podStartSLOduration=33.035760315 podStartE2EDuration="33.035760315s" podCreationTimestamp="2026-04-23 17:43:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:44:13.03530737 +0000 UTC m=+152.054760385" watchObservedRunningTime="2026-04-23 17:44:13.035760315 +0000 UTC m=+152.055213333" Apr 23 17:44:13.719444 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:13.719406 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:13.722153 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:13.722129 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:14.017487 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:14.017457 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:14.018882 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:14.018855 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-747f569c94-mjbpd" Apr 23 17:44:15.020816 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:15.020775 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" event={"ID":"eb0a83a2-e27a-459f-a491-f2ffb4073120","Type":"ContainerStarted","Data":"eacb5eac1918603f0174f2c1fb8d2b6f822b4404f15ed1c20742831f1abaf025"} Apr 23 17:44:15.043092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:15.043040 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-5cln8" podStartSLOduration=33.604681986 podStartE2EDuration="35.043026372s" podCreationTimestamp="2026-04-23 17:43:40 +0000 UTC" firstStartedPulling="2026-04-23 17:44:12.844095036 +0000 UTC m=+151.863548030" lastFinishedPulling="2026-04-23 17:44:14.282439418 +0000 UTC m=+153.301892416" observedRunningTime="2026-04-23 17:44:15.042371932 +0000 UTC m=+154.061824948" watchObservedRunningTime="2026-04-23 17:44:15.043026372 +0000 UTC m=+154.062479385" Apr 23 17:44:17.323858 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:44:17.323812 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9786k" podUID="3459e220-189c-4bb2-91fd-ebd60ad6899e" Apr 23 17:44:17.331977 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:44:17.331939 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-bwrpd" podUID="d2b26376-6f53-4edd-89f2-a4fe64523424" Apr 23 17:44:17.609632 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:44:17.609529 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-76492" podUID="eb1ab24a-5f1e-4324-9b5f-f92abb7647fa" Apr 23 17:44:18.029676 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:18.029647 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:44:18.029855 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:18.029653 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9786k" Apr 23 17:44:22.282481 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.282427 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:44:22.282481 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.282489 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:44:22.284784 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.284760 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3459e220-189c-4bb2-91fd-ebd60ad6899e-metrics-tls\") pod \"dns-default-9786k\" (UID: \"3459e220-189c-4bb2-91fd-ebd60ad6899e\") " pod="openshift-dns/dns-default-9786k" Apr 23 17:44:22.285184 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.285162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2b26376-6f53-4edd-89f2-a4fe64523424-cert\") pod \"ingress-canary-bwrpd\" (UID: \"d2b26376-6f53-4edd-89f2-a4fe64523424\") " pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:44:22.533104 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.533026 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-576jd\"" Apr 23 17:44:22.533391 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.533374 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zw6qq\"" Apr 23 17:44:22.540982 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.540963 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9786k" Apr 23 17:44:22.541061 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.541040 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bwrpd" Apr 23 17:44:22.690346 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.690315 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9786k"] Apr 23 17:44:22.693209 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:44:22.693181 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3459e220_189c_4bb2_91fd_ebd60ad6899e.slice/crio-9399139355d271bf8dd2e0f312232f4a1393a52301801bbd37a25dca0de3b8ab WatchSource:0}: Error finding container 9399139355d271bf8dd2e0f312232f4a1393a52301801bbd37a25dca0de3b8ab: Status 404 returned error can't find the container with id 9399139355d271bf8dd2e0f312232f4a1393a52301801bbd37a25dca0de3b8ab Apr 23 17:44:22.706043 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:22.706021 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bwrpd"] Apr 23 17:44:22.708758 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:44:22.708732 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b26376_6f53_4edd_89f2_a4fe64523424.slice/crio-c150a31429f9777b4c4a6f564425b3321b0fe6e7011bc07d62cf327bd86da2c6 WatchSource:0}: Error finding container c150a31429f9777b4c4a6f564425b3321b0fe6e7011bc07d62cf327bd86da2c6: Status 404 returned error can't find the container with id c150a31429f9777b4c4a6f564425b3321b0fe6e7011bc07d62cf327bd86da2c6 Apr 23 17:44:23.043413 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:23.043377 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9786k" event={"ID":"3459e220-189c-4bb2-91fd-ebd60ad6899e","Type":"ContainerStarted","Data":"9399139355d271bf8dd2e0f312232f4a1393a52301801bbd37a25dca0de3b8ab"} Apr 23 17:44:23.044319 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:23.044297 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bwrpd" event={"ID":"d2b26376-6f53-4edd-89f2-a4fe64523424","Type":"ContainerStarted","Data":"c150a31429f9777b4c4a6f564425b3321b0fe6e7011bc07d62cf327bd86da2c6"} Apr 23 17:44:25.050464 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:25.050419 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bwrpd" event={"ID":"d2b26376-6f53-4edd-89f2-a4fe64523424","Type":"ContainerStarted","Data":"62af3c66d96a24aba797d25a4e95546a824c6a587d959c14d4296afdf7a7e92d"} Apr 23 17:44:25.052064 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:25.052042 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9786k" event={"ID":"3459e220-189c-4bb2-91fd-ebd60ad6899e","Type":"ContainerStarted","Data":"036229e716d0c121feb8b953b112abb837f4a984260ce47fea1da46a93d12055"} Apr 23 17:44:25.052178 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:25.052073 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9786k" event={"ID":"3459e220-189c-4bb2-91fd-ebd60ad6899e","Type":"ContainerStarted","Data":"7683a09e307d31d1460304ca2d4b47030fb6263eb6432a0f0c51be3e7d32b202"} Apr 23 17:44:25.052178 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:25.052168 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9786k" Apr 23 17:44:25.069043 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:25.068999 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bwrpd" podStartSLOduration=128.828554233 podStartE2EDuration="2m11.06898589s" podCreationTimestamp="2026-04-23 17:42:14 +0000 UTC" firstStartedPulling="2026-04-23 17:44:22.710457108 +0000 UTC m=+161.729910101" lastFinishedPulling="2026-04-23 17:44:24.950888763 +0000 UTC m=+163.970341758" observedRunningTime="2026-04-23 17:44:25.068235244 +0000 UTC m=+164.087688262" watchObservedRunningTime="2026-04-23 17:44:25.06898589 +0000 UTC m=+164.088438908" Apr 23 17:44:25.086300 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:25.086248 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-9786k" podStartSLOduration=129.812411695 podStartE2EDuration="2m11.086236233s" podCreationTimestamp="2026-04-23 17:42:14 +0000 UTC" firstStartedPulling="2026-04-23 17:44:22.694979954 +0000 UTC m=+161.714432962" lastFinishedPulling="2026-04-23 17:44:23.968804507 +0000 UTC m=+162.988257500" observedRunningTime="2026-04-23 17:44:25.085539241 +0000 UTC m=+164.104992258" watchObservedRunningTime="2026-04-23 17:44:25.086236233 +0000 UTC m=+164.105689289" Apr 23 17:44:29.590489 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:29.590448 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:44:35.057377 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:44:35.057345 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9786k" Apr 23 17:45:02.619915 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:02.619886 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9786k_3459e220-189c-4bb2-91fd-ebd60ad6899e/dns/0.log" Apr 23 17:45:02.817063 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:02.817032 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9786k_3459e220-189c-4bb2-91fd-ebd60ad6899e/kube-rbac-proxy/0.log" Apr 23 17:45:03.817189 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:03.817161 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-728bd_bb810179-e96c-4f26-a7fc-e3006013faed/dns-node-resolver/0.log" Apr 23 17:45:05.217957 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:05.217932 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xlkpg_9a5b8fa2-c8ef-4328-b02c-5663e2246f76/node-ca/0.log" Apr 23 17:45:05.417948 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:05.417921 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-747f569c94-mjbpd_2dc66a6c-a16e-41b1-9fd0-cd602ca9e666/router/0.log" Apr 23 17:45:05.617333 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:05.617299 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bwrpd_d2b26376-6f53-4edd-89f2-a4fe64523424/serve-healthcheck-canary/0.log" Apr 23 17:45:06.218920 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:06.218895 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-7c6cbb6c87-qgzxg_5c365186-3cfd-4cd1-868f-72da2be8292b/volume-data-source-validator/0.log" Apr 23 17:45:06.417641 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:06.417614 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5bkht_a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1/csi-driver/0.log" Apr 23 17:45:06.618045 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:06.618015 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5bkht_a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1/csi-node-driver-registrar/0.log" Apr 23 17:45:06.818287 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:06.818252 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5bkht_a20ca6c6-f757-4ca7-ae17-4c2f1774e1e1/csi-liveness-probe/0.log" Apr 23 17:45:09.158112 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:09.158079 2575 generic.go:358] "Generic (PLEG): container finished" podID="66e27305-61d1-45ca-a35f-ce823c1e6435" containerID="5df342216a541a46898516aa0cd3e92db461bc28369f674e979b4b0f1fd96fd2" exitCode=0 Apr 23 17:45:09.158645 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:09.158136 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" event={"ID":"66e27305-61d1-45ca-a35f-ce823c1e6435","Type":"ContainerDied","Data":"5df342216a541a46898516aa0cd3e92db461bc28369f674e979b4b0f1fd96fd2"} Apr 23 17:45:09.158645 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:09.158484 2575 scope.go:117] "RemoveContainer" containerID="5df342216a541a46898516aa0cd3e92db461bc28369f674e979b4b0f1fd96fd2" Apr 23 17:45:10.162303 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:10.162271 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-fc8h5" event={"ID":"66e27305-61d1-45ca-a35f-ce823c1e6435","Type":"ContainerStarted","Data":"6a7017d8a8b0af924a27865d5916eef801959db242c9a627e6e4d02ad81324ea"} Apr 23 17:45:53.330827 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:53.330789 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:45:53.333053 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:53.333035 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1ab24a-5f1e-4324-9b5f-f92abb7647fa-metrics-certs\") pod \"network-metrics-daemon-76492\" (UID: \"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa\") " pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:45:53.594038 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:53.593731 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wdbsc\"" Apr 23 17:45:53.601414 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:53.601390 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76492" Apr 23 17:45:53.716323 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:53.716291 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-76492"] Apr 23 17:45:53.720137 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:45:53.720110 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1ab24a_5f1e_4324_9b5f_f92abb7647fa.slice/crio-be86ccfa755586ccd9cb9728c4c8ae274e540ddbdac1cb4bd48446c613c1a92b WatchSource:0}: Error finding container be86ccfa755586ccd9cb9728c4c8ae274e540ddbdac1cb4bd48446c613c1a92b: Status 404 returned error can't find the container with id be86ccfa755586ccd9cb9728c4c8ae274e540ddbdac1cb4bd48446c613c1a92b Apr 23 17:45:54.269099 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:54.269015 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-76492" event={"ID":"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa","Type":"ContainerStarted","Data":"be86ccfa755586ccd9cb9728c4c8ae274e540ddbdac1cb4bd48446c613c1a92b"} Apr 23 17:45:55.273219 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:55.273177 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-76492" event={"ID":"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa","Type":"ContainerStarted","Data":"363ac7274624b43a7c218debec7cb6f1013656109ede396c0a5c08a73d1f87fb"} Apr 23 17:45:55.273219 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:55.273220 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-76492" event={"ID":"eb1ab24a-5f1e-4324-9b5f-f92abb7647fa","Type":"ContainerStarted","Data":"89b3ce1abdc8d98ce82af01b55fd6d4337f03bbd2787b286f40a17ce4887955a"} Apr 23 17:45:55.289386 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:45:55.289327 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-76492" podStartSLOduration=253.396311163 podStartE2EDuration="4m14.289311571s" podCreationTimestamp="2026-04-23 17:41:41 +0000 UTC" firstStartedPulling="2026-04-23 17:45:53.722320921 +0000 UTC m=+252.741773915" lastFinishedPulling="2026-04-23 17:45:54.615321329 +0000 UTC m=+253.634774323" observedRunningTime="2026-04-23 17:45:55.289206106 +0000 UTC m=+254.308659160" watchObservedRunningTime="2026-04-23 17:45:55.289311571 +0000 UTC m=+254.308764594" Apr 23 17:46:41.431878 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:41.431840 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:46:41.432728 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:41.432707 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:46:42.581248 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.581211 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wsvkb"] Apr 23 17:46:42.585095 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.585068 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.589097 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.589029 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 23 17:46:42.589256 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.589142 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 23 17:46:42.589613 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.589567 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9sps\"" Apr 23 17:46:42.589711 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.589608 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 23 17:46:42.591079 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.591056 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 23 17:46:42.598987 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.598963 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wsvkb"] Apr 23 17:46:42.648866 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.648826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a44b595-de5f-4334-bb31-1c5aa5d798a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.648866 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.648866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gwc\" (UniqueName: \"kubernetes.io/projected/0a44b595-de5f-4334-bb31-1c5aa5d798a2-kube-api-access-24gwc\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.649072 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.648901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a44b595-de5f-4334-bb31-1c5aa5d798a2-data-volume\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.649072 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.648954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a44b595-de5f-4334-bb31-1c5aa5d798a2-crio-socket\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.649072 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.649002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a44b595-de5f-4334-bb31-1c5aa5d798a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750089 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750054 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a44b595-de5f-4334-bb31-1c5aa5d798a2-data-volume\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750089 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750093 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a44b595-de5f-4334-bb31-1c5aa5d798a2-crio-socket\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750359 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750112 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a44b595-de5f-4334-bb31-1c5aa5d798a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750359 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750152 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a44b595-de5f-4334-bb31-1c5aa5d798a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750359 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750186 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-24gwc\" (UniqueName: \"kubernetes.io/projected/0a44b595-de5f-4334-bb31-1c5aa5d798a2-kube-api-access-24gwc\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750359 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750212 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a44b595-de5f-4334-bb31-1c5aa5d798a2-crio-socket\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.750552 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.750488 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a44b595-de5f-4334-bb31-1c5aa5d798a2-data-volume\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.751140 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.751120 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a44b595-de5f-4334-bb31-1c5aa5d798a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.752632 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.752607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a44b595-de5f-4334-bb31-1c5aa5d798a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.767702 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.767671 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gwc\" (UniqueName: \"kubernetes.io/projected/0a44b595-de5f-4334-bb31-1c5aa5d798a2-kube-api-access-24gwc\") pod \"insights-runtime-extractor-wsvkb\" (UID: \"0a44b595-de5f-4334-bb31-1c5aa5d798a2\") " pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:42.895249 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:42.895151 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wsvkb" Apr 23 17:46:43.029135 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:43.029110 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wsvkb"] Apr 23 17:46:43.032407 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:46:43.032376 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a44b595_de5f_4334_bb31_1c5aa5d798a2.slice/crio-8f869c0361cd914fa8d6fba6c9737119523de510e80f2430641b5032095dab41 WatchSource:0}: Error finding container 8f869c0361cd914fa8d6fba6c9737119523de510e80f2430641b5032095dab41: Status 404 returned error can't find the container with id 8f869c0361cd914fa8d6fba6c9737119523de510e80f2430641b5032095dab41 Apr 23 17:46:43.034307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:43.034287 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 17:46:43.397842 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:43.397806 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsvkb" event={"ID":"0a44b595-de5f-4334-bb31-1c5aa5d798a2","Type":"ContainerStarted","Data":"34a8bcf36e38364b9d699d73841b6258736f466fa1b253f2b61d1f90305620e1"} Apr 23 17:46:43.397842 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:43.397842 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsvkb" event={"ID":"0a44b595-de5f-4334-bb31-1c5aa5d798a2","Type":"ContainerStarted","Data":"8f869c0361cd914fa8d6fba6c9737119523de510e80f2430641b5032095dab41"} Apr 23 17:46:44.402888 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:44.402846 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsvkb" event={"ID":"0a44b595-de5f-4334-bb31-1c5aa5d798a2","Type":"ContainerStarted","Data":"a8a00feab12f5edf1f15f406a350d4a9ef812d931e360cbbeb11f6c1d7b81864"} Apr 23 17:46:45.407427 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:45.407343 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsvkb" event={"ID":"0a44b595-de5f-4334-bb31-1c5aa5d798a2","Type":"ContainerStarted","Data":"a323b1ee5b03c725a7defddae4308f6bb74f680fa23602ccb8746ad7b349826d"} Apr 23 17:46:45.459133 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:45.459084 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wsvkb" podStartSLOduration=1.428436199 podStartE2EDuration="3.459068927s" podCreationTimestamp="2026-04-23 17:46:42 +0000 UTC" firstStartedPulling="2026-04-23 17:46:43.102116747 +0000 UTC m=+302.121569741" lastFinishedPulling="2026-04-23 17:46:45.132749475 +0000 UTC m=+304.152202469" observedRunningTime="2026-04-23 17:46:45.457937771 +0000 UTC m=+304.477390787" watchObservedRunningTime="2026-04-23 17:46:45.459068927 +0000 UTC m=+304.478521943" Apr 23 17:46:49.804453 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.804415 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4jj65"] Apr 23 17:46:49.808247 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.808219 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.811227 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.811180 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 23 17:46:49.811952 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.811930 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hb422\"" Apr 23 17:46:49.812497 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.812456 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 23 17:46:49.812918 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.812902 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 23 17:46:49.818324 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.818306 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 23 17:46:49.901303 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-textfile\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901303 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-tls\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901543 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901543 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-root\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901543 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-wtmp\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901543 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a209846-480f-458a-94ef-f0de3126b8a5-metrics-client-ca\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901543 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901808 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfcl\" (UniqueName: \"kubernetes.io/projected/7a209846-480f-458a-94ef-f0de3126b8a5-kube-api-access-wlfcl\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:49.901808 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:49.901704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-sys\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002181 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002140 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a209846-480f-458a-94ef-f0de3126b8a5-metrics-client-ca\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002191 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002230 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfcl\" (UniqueName: \"kubernetes.io/projected/7a209846-480f-458a-94ef-f0de3126b8a5-kube-api-access-wlfcl\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002260 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-sys\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002294 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-textfile\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002322 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-tls\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002348 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002362 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-sys\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002398 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002378 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-root\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002823 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002428 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-root\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002823 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002463 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-wtmp\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002823 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-wtmp\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.002823 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-textfile\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.003026 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002819 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a209846-480f-458a-94ef-f0de3126b8a5-metrics-client-ca\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.003026 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.002998 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.004879 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.004856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-tls\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.005277 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.005256 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a209846-480f-458a-94ef-f0de3126b8a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.017120 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.017085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfcl\" (UniqueName: \"kubernetes.io/projected/7a209846-480f-458a-94ef-f0de3126b8a5-kube-api-access-wlfcl\") pod \"node-exporter-4jj65\" (UID: \"7a209846-480f-458a-94ef-f0de3126b8a5\") " pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.120159 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.120058 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jj65" Apr 23 17:46:50.129100 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:46:50.129052 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a209846_480f_458a_94ef_f0de3126b8a5.slice/crio-91e7f19a47cdafa4052469466bc59b18666de37e48723309705be45c19de4b15 WatchSource:0}: Error finding container 91e7f19a47cdafa4052469466bc59b18666de37e48723309705be45c19de4b15: Status 404 returned error can't find the container with id 91e7f19a47cdafa4052469466bc59b18666de37e48723309705be45c19de4b15 Apr 23 17:46:50.421952 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.421861 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jj65" event={"ID":"7a209846-480f-458a-94ef-f0de3126b8a5","Type":"ContainerStarted","Data":"91e7f19a47cdafa4052469466bc59b18666de37e48723309705be45c19de4b15"} Apr 23 17:46:50.754397 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.754360 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:46:50.759114 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.759083 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.763782 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.763753 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 23 17:46:50.768934 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.768906 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 23 17:46:50.769467 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.769446 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 23 17:46:50.769580 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.769526 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 23 17:46:50.769580 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.769559 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 23 17:46:50.770142 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.769913 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 23 17:46:50.770142 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.769975 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 23 17:46:50.770142 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.770001 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 23 17:46:50.770142 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.770052 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 23 17:46:50.770142 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.769979 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9z6nr\"" Apr 23 17:46:50.788899 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.788867 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:46:50.810154 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810154 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-config-volume\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-config-out\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456gc\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-kube-api-access-456gc\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-web-config\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.810581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.810579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.911827 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911782 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.911827 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911826 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911854 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911876 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-config-volume\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911904 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911924 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911947 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-config-out\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911969 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.911989 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.912016 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-456gc\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-kube-api-access-456gc\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.912047 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-web-config\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912087 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.912073 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912616 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.912123 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.912683 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.912611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.913382 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.912927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.913382 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.913132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.914978 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.914947 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.915291 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.915265 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-config-out\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.915426 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.915379 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.915671 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.915643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.915754 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.915684 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-web-config\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.915754 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.915691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.916006 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.915990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.916074 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.916050 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.916623 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.916582 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-config-volume\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:50.925507 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:50.925488 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456gc\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-kube-api-access-456gc\") pod \"alertmanager-main-0\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:51.071545 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:51.071505 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:46:51.262773 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:51.262703 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:46:51.264964 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:46:51.264932 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64df7d2e_90c8_499a_b312_23d8393445cd.slice/crio-1ae2ad16b2c1733ba28dcbd1da8fab2173550b570641cf1d215134f0fccb8669 WatchSource:0}: Error finding container 1ae2ad16b2c1733ba28dcbd1da8fab2173550b570641cf1d215134f0fccb8669: Status 404 returned error can't find the container with id 1ae2ad16b2c1733ba28dcbd1da8fab2173550b570641cf1d215134f0fccb8669 Apr 23 17:46:51.426750 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:51.426663 2575 generic.go:358] "Generic (PLEG): container finished" podID="7a209846-480f-458a-94ef-f0de3126b8a5" containerID="b37d727beac32868fe03eebe5a2e3270dad13414d5a151c5825bfbf567287630" exitCode=0 Apr 23 17:46:51.426750 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:51.426733 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jj65" event={"ID":"7a209846-480f-458a-94ef-f0de3126b8a5","Type":"ContainerDied","Data":"b37d727beac32868fe03eebe5a2e3270dad13414d5a151c5825bfbf567287630"} Apr 23 17:46:51.427800 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:51.427772 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"1ae2ad16b2c1733ba28dcbd1da8fab2173550b570641cf1d215134f0fccb8669"} Apr 23 17:46:52.432925 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:52.432881 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jj65" event={"ID":"7a209846-480f-458a-94ef-f0de3126b8a5","Type":"ContainerStarted","Data":"5b25ba76f1aa21e93837f3cb2f941ec3857bd9bf4b811c7fdfd4f5cfeb51e5c9"} Apr 23 17:46:52.432925 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:52.432929 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jj65" event={"ID":"7a209846-480f-458a-94ef-f0de3126b8a5","Type":"ContainerStarted","Data":"6cb4952c3e5b4e75e668602373777a8452e9786be6d155e7d33e82a7840e2357"} Apr 23 17:46:52.434169 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:52.434146 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a" exitCode=0 Apr 23 17:46:52.434263 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:52.434212 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a"} Apr 23 17:46:52.464617 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:52.464403 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4jj65" podStartSLOduration=2.7699972170000002 podStartE2EDuration="3.464386656s" podCreationTimestamp="2026-04-23 17:46:49 +0000 UTC" firstStartedPulling="2026-04-23 17:46:50.130784361 +0000 UTC m=+309.150237354" lastFinishedPulling="2026-04-23 17:46:50.82517379 +0000 UTC m=+309.844626793" observedRunningTime="2026-04-23 17:46:52.464070042 +0000 UTC m=+311.483523059" watchObservedRunningTime="2026-04-23 17:46:52.464386656 +0000 UTC m=+311.483839673" Apr 23 17:46:54.443062 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.443022 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde"} Apr 23 17:46:54.443062 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.443064 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807"} Apr 23 17:46:54.443502 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.443076 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62"} Apr 23 17:46:54.443502 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.443086 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6"} Apr 23 17:46:54.443502 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.443095 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7"} Apr 23 17:46:54.694489 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.694423 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-676bffff47-9vj52"] Apr 23 17:46:54.697579 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.697563 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.701788 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.701768 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Apr 23 17:46:54.702993 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.702973 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Apr 23 17:46:54.703368 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.703350 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Apr 23 17:46:54.703819 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.703739 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Apr 23 17:46:54.704064 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.703830 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66cmr\"" Apr 23 17:46:54.704064 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.703740 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Apr 23 17:46:54.704064 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.703748 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Apr 23 17:46:54.717497 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.717477 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Apr 23 17:46:54.718923 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.718904 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Apr 23 17:46:54.728575 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.728553 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676bffff47-9vj52"] Apr 23 17:46:54.746567 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-oauth-config\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.746670 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-oauth-serving-cert\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.746670 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-serving-cert\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.746670 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-config\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.746775 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-service-ca\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.746775 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-trusted-ca-bundle\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.746775 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.746768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8lp\" (UniqueName: \"kubernetes.io/projected/d70646d2-2b34-443c-aad1-c46cf2eac4bd-kube-api-access-vc8lp\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.847653 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847621 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-oauth-config\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.847797 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847658 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-oauth-serving-cert\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.847797 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847683 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-serving-cert\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.847797 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847705 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-config\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.847797 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847737 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-service-ca\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.847797 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847769 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-trusted-ca-bundle\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.848041 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.847813 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8lp\" (UniqueName: \"kubernetes.io/projected/d70646d2-2b34-443c-aad1-c46cf2eac4bd-kube-api-access-vc8lp\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.848448 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.848417 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-oauth-serving-cert\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.848564 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.848545 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-service-ca\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.848564 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.848553 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-config\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.848837 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.848812 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-trusted-ca-bundle\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.850474 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.850457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-serving-cert\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.850554 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.850541 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-oauth-config\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:54.865840 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:54.865818 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8lp\" (UniqueName: \"kubernetes.io/projected/d70646d2-2b34-443c-aad1-c46cf2eac4bd-kube-api-access-vc8lp\") pod \"console-676bffff47-9vj52\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:55.006098 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:55.006058 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:46:55.143060 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:55.143031 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676bffff47-9vj52"] Apr 23 17:46:55.147488 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:46:55.147459 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70646d2_2b34_443c_aad1_c46cf2eac4bd.slice/crio-329c8d6508065f79466f398d3f41fb6926a4383a37045f963d5a5d696adeb498 WatchSource:0}: Error finding container 329c8d6508065f79466f398d3f41fb6926a4383a37045f963d5a5d696adeb498: Status 404 returned error can't find the container with id 329c8d6508065f79466f398d3f41fb6926a4383a37045f963d5a5d696adeb498 Apr 23 17:46:55.446885 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:55.446795 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676bffff47-9vj52" event={"ID":"d70646d2-2b34-443c-aad1-c46cf2eac4bd","Type":"ContainerStarted","Data":"329c8d6508065f79466f398d3f41fb6926a4383a37045f963d5a5d696adeb498"} Apr 23 17:46:55.449545 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:55.449517 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerStarted","Data":"c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a"} Apr 23 17:46:55.505421 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:55.505374 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.174542523 podStartE2EDuration="5.505361061s" podCreationTimestamp="2026-04-23 17:46:50 +0000 UTC" firstStartedPulling="2026-04-23 17:46:51.26691647 +0000 UTC m=+310.286369463" lastFinishedPulling="2026-04-23 17:46:54.597735002 +0000 UTC m=+313.617188001" observedRunningTime="2026-04-23 17:46:55.504333809 +0000 UTC m=+314.523786824" watchObservedRunningTime="2026-04-23 17:46:55.505361061 +0000 UTC m=+314.524814077" Apr 23 17:46:58.460423 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:58.460385 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676bffff47-9vj52" event={"ID":"d70646d2-2b34-443c-aad1-c46cf2eac4bd","Type":"ContainerStarted","Data":"0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7"} Apr 23 17:46:58.483086 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:58.483034 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-676bffff47-9vj52" podStartSLOduration=2.00328443 podStartE2EDuration="4.483020064s" podCreationTimestamp="2026-04-23 17:46:54 +0000 UTC" firstStartedPulling="2026-04-23 17:46:55.149298162 +0000 UTC m=+314.168751156" lastFinishedPulling="2026-04-23 17:46:57.629033779 +0000 UTC m=+316.648486790" observedRunningTime="2026-04-23 17:46:58.481793624 +0000 UTC m=+317.501246641" watchObservedRunningTime="2026-04-23 17:46:58.483020064 +0000 UTC m=+317.502473091" Apr 23 17:46:59.581204 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:46:59.581168 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676bffff47-9vj52"] Apr 23 17:47:05.006799 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:05.006766 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:47:25.485156 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.485027 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-676bffff47-9vj52" podUID="d70646d2-2b34-443c-aad1-c46cf2eac4bd" containerName="console" containerID="cri-o://0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7" gracePeriod=15 Apr 23 17:47:25.724212 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.724190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-676bffff47-9vj52_d70646d2-2b34-443c-aad1-c46cf2eac4bd/console/0.log" Apr 23 17:47:25.724312 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.724261 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:47:25.886469 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886381 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-trusted-ca-bundle\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886469 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886464 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-oauth-serving-cert\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886729 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886497 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-oauth-config\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886729 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886535 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-serving-cert\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886729 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886562 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc8lp\" (UniqueName: \"kubernetes.io/projected/d70646d2-2b34-443c-aad1-c46cf2eac4bd-kube-api-access-vc8lp\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886729 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886624 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-config\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886729 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886658 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-service-ca\") pod \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\" (UID: \"d70646d2-2b34-443c-aad1-c46cf2eac4bd\") " Apr 23 17:47:25.886729 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886666 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:47:25.887017 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886839 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-trusted-ca-bundle\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:25.887017 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.886926 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:47:25.887090 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.887028 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-config" (OuterVolumeSpecName: "console-config") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:47:25.887148 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.887123 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:47:25.888829 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.888803 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:47:25.888829 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.888805 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70646d2-2b34-443c-aad1-c46cf2eac4bd-kube-api-access-vc8lp" (OuterVolumeSpecName: "kube-api-access-vc8lp") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "kube-api-access-vc8lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 17:47:25.888829 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.888820 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d70646d2-2b34-443c-aad1-c46cf2eac4bd" (UID: "d70646d2-2b34-443c-aad1-c46cf2eac4bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:47:25.987621 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.987570 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-oauth-serving-cert\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:25.987621 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.987616 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-oauth-config\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:25.987621 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.987625 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-serving-cert\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:25.987819 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.987634 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc8lp\" (UniqueName: \"kubernetes.io/projected/d70646d2-2b34-443c-aad1-c46cf2eac4bd-kube-api-access-vc8lp\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:25.987819 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.987644 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-console-config\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:25.987819 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:25.987653 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d70646d2-2b34-443c-aad1-c46cf2eac4bd-service-ca\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:47:26.532404 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.532379 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-676bffff47-9vj52_d70646d2-2b34-443c-aad1-c46cf2eac4bd/console/0.log" Apr 23 17:47:26.532886 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.532417 2575 generic.go:358] "Generic (PLEG): container finished" podID="d70646d2-2b34-443c-aad1-c46cf2eac4bd" containerID="0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7" exitCode=2 Apr 23 17:47:26.532886 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.532503 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676bffff47-9vj52" Apr 23 17:47:26.532886 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.532501 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676bffff47-9vj52" event={"ID":"d70646d2-2b34-443c-aad1-c46cf2eac4bd","Type":"ContainerDied","Data":"0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7"} Apr 23 17:47:26.532886 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.532628 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676bffff47-9vj52" event={"ID":"d70646d2-2b34-443c-aad1-c46cf2eac4bd","Type":"ContainerDied","Data":"329c8d6508065f79466f398d3f41fb6926a4383a37045f963d5a5d696adeb498"} Apr 23 17:47:26.532886 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.532644 2575 scope.go:117] "RemoveContainer" containerID="0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7" Apr 23 17:47:26.540939 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.540923 2575 scope.go:117] "RemoveContainer" containerID="0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7" Apr 23 17:47:26.541188 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:47:26.541167 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7\": container with ID starting with 0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7 not found: ID does not exist" containerID="0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7" Apr 23 17:47:26.541242 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.541196 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7"} err="failed to get container status \"0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7\": rpc error: code = NotFound desc = could not find container \"0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7\": container with ID starting with 0a2556396b82677bf30ae62912c3814b3aa09fe3e632a11e90748a5956eeb8a7 not found: ID does not exist" Apr 23 17:47:26.553855 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.553830 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676bffff47-9vj52"] Apr 23 17:47:26.558305 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:26.558284 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-676bffff47-9vj52"] Apr 23 17:47:27.594283 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:47:27.594238 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70646d2-2b34-443c-aad1-c46cf2eac4bd" path="/var/lib/kubelet/pods/d70646d2-2b34-443c-aad1-c46cf2eac4bd/volumes" Apr 23 17:48:06.900945 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.900866 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c54c7699-h72c2"] Apr 23 17:48:06.901581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.901483 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70646d2-2b34-443c-aad1-c46cf2eac4bd" containerName="console" Apr 23 17:48:06.901581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.901505 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70646d2-2b34-443c-aad1-c46cf2eac4bd" containerName="console" Apr 23 17:48:06.901754 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.901616 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="d70646d2-2b34-443c-aad1-c46cf2eac4bd" containerName="console" Apr 23 17:48:06.903846 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.903825 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.906770 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906746 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Apr 23 17:48:06.906891 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906786 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Apr 23 17:48:06.906891 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906800 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Apr 23 17:48:06.906891 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906876 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Apr 23 17:48:06.907042 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906899 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Apr 23 17:48:06.907042 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906898 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Apr 23 17:48:06.907042 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.906961 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66cmr\"" Apr 23 17:48:06.909514 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.909497 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Apr 23 17:48:06.914669 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.914649 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Apr 23 17:48:06.918702 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.918683 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c54c7699-h72c2"] Apr 23 17:48:06.972229 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972192 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-console-config\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.972229 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-service-ca\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.972426 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-oauth-config\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.972426 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-oauth-serving-cert\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.972426 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnjj\" (UniqueName: \"kubernetes.io/projected/91a79641-ed93-4060-82ee-6a919793a7a4-kube-api-access-jrnjj\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.972426 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-serving-cert\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:06.972426 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:06.972390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-trusted-ca-bundle\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073575 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073536 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-oauth-serving-cert\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073575 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073570 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnjj\" (UniqueName: \"kubernetes.io/projected/91a79641-ed93-4060-82ee-6a919793a7a4-kube-api-access-jrnjj\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073804 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073628 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-serving-cert\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073804 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073656 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-trusted-ca-bundle\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073804 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073781 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-console-config\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073915 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073824 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-service-ca\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.073915 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.073868 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-oauth-config\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.074339 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.074281 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-oauth-serving-cert\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.074468 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.074453 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-console-config\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.074650 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.074625 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-service-ca\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.074704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.074647 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-trusted-ca-bundle\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.076069 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.076043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-oauth-config\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.076160 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.076142 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-serving-cert\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.082423 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.082397 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnjj\" (UniqueName: \"kubernetes.io/projected/91a79641-ed93-4060-82ee-6a919793a7a4-kube-api-access-jrnjj\") pod \"console-69c54c7699-h72c2\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.213975 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.213884 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:07.331453 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.331332 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c54c7699-h72c2"] Apr 23 17:48:07.334230 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:48:07.334199 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a79641_ed93_4060_82ee_6a919793a7a4.slice/crio-f1c1eb57e94427e11f522259cff896f07c01ad9e1b6af7e74fe863f157f2a8a2 WatchSource:0}: Error finding container f1c1eb57e94427e11f522259cff896f07c01ad9e1b6af7e74fe863f157f2a8a2: Status 404 returned error can't find the container with id f1c1eb57e94427e11f522259cff896f07c01ad9e1b6af7e74fe863f157f2a8a2 Apr 23 17:48:07.636980 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.636947 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c54c7699-h72c2" event={"ID":"91a79641-ed93-4060-82ee-6a919793a7a4","Type":"ContainerStarted","Data":"97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268"} Apr 23 17:48:07.636980 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:07.636983 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c54c7699-h72c2" event={"ID":"91a79641-ed93-4060-82ee-6a919793a7a4","Type":"ContainerStarted","Data":"f1c1eb57e94427e11f522259cff896f07c01ad9e1b6af7e74fe863f157f2a8a2"} Apr 23 17:48:10.302454 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.302398 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-69c54c7699-h72c2" podStartSLOduration=4.302384214 podStartE2EDuration="4.302384214s" podCreationTimestamp="2026-04-23 17:48:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:48:07.660804534 +0000 UTC m=+386.680257549" watchObservedRunningTime="2026-04-23 17:48:10.302384214 +0000 UTC m=+389.321837231" Apr 23 17:48:10.303556 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.303535 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:48:10.303982 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.303960 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="alertmanager" containerID="cri-o://4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7" gracePeriod=120 Apr 23 17:48:10.304233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.304041 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-metric" containerID="cri-o://4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde" gracePeriod=120 Apr 23 17:48:10.304233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.304042 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-web" containerID="cri-o://090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62" gracePeriod=120 Apr 23 17:48:10.304233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.304063 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy" containerID="cri-o://0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807" gracePeriod=120 Apr 23 17:48:10.304233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.304111 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="prom-label-proxy" containerID="cri-o://c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a" gracePeriod=120 Apr 23 17:48:10.305076 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.304570 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="config-reloader" containerID="cri-o://e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6" gracePeriod=120 Apr 23 17:48:10.648232 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648148 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a" exitCode=0 Apr 23 17:48:10.648232 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648173 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807" exitCode=0 Apr 23 17:48:10.648232 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648180 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62" exitCode=0 Apr 23 17:48:10.648232 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648185 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6" exitCode=0 Apr 23 17:48:10.648232 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648192 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7" exitCode=0 Apr 23 17:48:10.648232 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648215 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a"} Apr 23 17:48:10.648520 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648247 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807"} Apr 23 17:48:10.648520 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648257 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62"} Apr 23 17:48:10.648520 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648266 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6"} Apr 23 17:48:10.648520 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:10.648275 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7"} Apr 23 17:48:11.565636 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.565613 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.608572 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608548 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-main-tls\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608743 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608610 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-config-volume\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608743 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608637 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608743 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608677 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-cluster-tls-config\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608743 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608706 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-web-config\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608741 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-trusted-ca-bundle\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608779 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-main-db\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608803 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-tls-assets\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608842 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-config-out\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608876 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-456gc\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-kube-api-access-456gc\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608920 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.608955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608947 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.609322 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.608974 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-metrics-client-ca\") pod \"64df7d2e-90c8-499a-b312-23d8393445cd\" (UID: \"64df7d2e-90c8-499a-b312-23d8393445cd\") " Apr 23 17:48:11.609407 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.609291 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:48:11.609481 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.609459 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:48:11.610281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.609751 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 17:48:11.611845 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.611812 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-kube-api-access-456gc" (OuterVolumeSpecName: "kube-api-access-456gc") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "kube-api-access-456gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 17:48:11.613631 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.612272 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.613631 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.612504 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-config-out" (OuterVolumeSpecName: "config-out") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 17:48:11.614198 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.614154 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.614356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.614320 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.614456 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.614399 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.614866 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.614841 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-config-volume" (OuterVolumeSpecName: "config-volume") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.616249 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.616225 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 17:48:11.617195 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.617059 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.623544 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.623480 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-web-config" (OuterVolumeSpecName: "web-config") pod "64df7d2e-90c8-499a-b312-23d8393445cd" (UID: "64df7d2e-90c8-499a-b312-23d8393445cd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:48:11.653184 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.653158 2575 generic.go:358] "Generic (PLEG): container finished" podID="64df7d2e-90c8-499a-b312-23d8393445cd" containerID="4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde" exitCode=0 Apr 23 17:48:11.653287 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.653250 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde"} Apr 23 17:48:11.653287 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.653263 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.653287 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.653275 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df7d2e-90c8-499a-b312-23d8393445cd","Type":"ContainerDied","Data":"1ae2ad16b2c1733ba28dcbd1da8fab2173550b570641cf1d215134f0fccb8669"} Apr 23 17:48:11.653387 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.653290 2575 scope.go:117] "RemoveContainer" containerID="c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a" Apr 23 17:48:11.659987 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.659971 2575 scope.go:117] "RemoveContainer" containerID="4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde" Apr 23 17:48:11.666298 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.666281 2575 scope.go:117] "RemoveContainer" containerID="0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807" Apr 23 17:48:11.672022 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.672000 2575 scope.go:117] "RemoveContainer" containerID="090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62" Apr 23 17:48:11.677292 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.677270 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:48:11.678358 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.678340 2575 scope.go:117] "RemoveContainer" containerID="e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6" Apr 23 17:48:11.682959 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.682938 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:48:11.684855 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.684837 2575 scope.go:117] "RemoveContainer" containerID="4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7" Apr 23 17:48:11.690746 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.690732 2575 scope.go:117] "RemoveContainer" containerID="9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a" Apr 23 17:48:11.697888 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.697869 2575 scope.go:117] "RemoveContainer" containerID="c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a" Apr 23 17:48:11.698146 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.698128 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a\": container with ID starting with c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a not found: ID does not exist" containerID="c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a" Apr 23 17:48:11.698198 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.698158 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a"} err="failed to get container status \"c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a\": rpc error: code = NotFound desc = could not find container \"c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a\": container with ID starting with c2a54c93e34ecd2e927b96de7d249f0ef8038d251d9fd4483f0ad463100bb24a not found: ID does not exist" Apr 23 17:48:11.698198 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.698175 2575 scope.go:117] "RemoveContainer" containerID="4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde" Apr 23 17:48:11.698417 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.698401 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde\": container with ID starting with 4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde not found: ID does not exist" containerID="4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde" Apr 23 17:48:11.698453 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.698424 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde"} err="failed to get container status \"4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde\": rpc error: code = NotFound desc = could not find container \"4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde\": container with ID starting with 4ba9c29d485da02d1cd94d31d3f11f4307e5e1e8c667fb7cd334a71398771bde not found: ID does not exist" Apr 23 17:48:11.698453 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.698441 2575 scope.go:117] "RemoveContainer" containerID="0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807" Apr 23 17:48:11.698757 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.698737 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807\": container with ID starting with 0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807 not found: ID does not exist" containerID="0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807" Apr 23 17:48:11.698807 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.698764 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807"} err="failed to get container status \"0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807\": rpc error: code = NotFound desc = could not find container \"0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807\": container with ID starting with 0736ae8f38f0f1464eed49bcfabc7fa330807c03c304f2b0f0cb79e36189a807 not found: ID does not exist" Apr 23 17:48:11.698807 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.698778 2575 scope.go:117] "RemoveContainer" containerID="090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62" Apr 23 17:48:11.699066 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.699050 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62\": container with ID starting with 090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62 not found: ID does not exist" containerID="090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62" Apr 23 17:48:11.699115 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699069 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62"} err="failed to get container status \"090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62\": rpc error: code = NotFound desc = could not find container \"090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62\": container with ID starting with 090d0c706d368d83c87d640583d17da8b7491daca0b15c0deb952470aae76e62 not found: ID does not exist" Apr 23 17:48:11.699115 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699081 2575 scope.go:117] "RemoveContainer" containerID="e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6" Apr 23 17:48:11.699305 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.699289 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6\": container with ID starting with e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6 not found: ID does not exist" containerID="e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6" Apr 23 17:48:11.699342 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699313 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6"} err="failed to get container status \"e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6\": rpc error: code = NotFound desc = could not find container \"e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6\": container with ID starting with e55ad83f86953ff18148320e174aa44c6fc4e0c1c752291fe36f2ac51a7699e6 not found: ID does not exist" Apr 23 17:48:11.699342 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699327 2575 scope.go:117] "RemoveContainer" containerID="4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7" Apr 23 17:48:11.699546 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.699531 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7\": container with ID starting with 4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7 not found: ID does not exist" containerID="4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7" Apr 23 17:48:11.699608 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699548 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7"} err="failed to get container status \"4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7\": rpc error: code = NotFound desc = could not find container \"4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7\": container with ID starting with 4549f398143b1317923aeaf3ac464cc2b8d01b44c87b09a7771892fe851d30f7 not found: ID does not exist" Apr 23 17:48:11.699608 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699561 2575 scope.go:117] "RemoveContainer" containerID="9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a" Apr 23 17:48:11.699818 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:48:11.699799 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a\": container with ID starting with 9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a not found: ID does not exist" containerID="9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a" Apr 23 17:48:11.699863 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.699822 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a"} err="failed to get container status \"9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a\": rpc error: code = NotFound desc = could not find container \"9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a\": container with ID starting with 9defaef23be115aa99805d8791ed2d3ebe7d52793bbd02eb6050d85de51db60a not found: ID does not exist" Apr 23 17:48:11.710196 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710175 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-cluster-tls-config\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710196 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710194 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-web-config\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710204 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710213 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-alertmanager-main-db\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710223 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-tls-assets\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710234 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df7d2e-90c8-499a-b312-23d8393445cd-config-out\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710246 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-456gc\" (UniqueName: \"kubernetes.io/projected/64df7d2e-90c8-499a-b312-23d8393445cd-kube-api-access-456gc\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710257 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710278 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710289 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df7d2e-90c8-499a-b312-23d8393445cd-metrics-client-ca\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710298 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-main-tls\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710307 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710306 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-config-volume\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.710605 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.710319 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df7d2e-90c8-499a-b312-23d8393445cd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:48:11.714376 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714357 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:48:11.714655 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714641 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy" Apr 23 17:48:11.714704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714657 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy" Apr 23 17:48:11.714704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714668 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-metric" Apr 23 17:48:11.714704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714674 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-metric" Apr 23 17:48:11.714704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714682 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="config-reloader" Apr 23 17:48:11.714704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714690 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="config-reloader" Apr 23 17:48:11.714704 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714700 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="prom-label-proxy" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714708 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="prom-label-proxy" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714720 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="init-config-reloader" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714728 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="init-config-reloader" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714736 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-web" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714744 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-web" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714752 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="alertmanager" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714759 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="alertmanager" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714814 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714827 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-metric" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714835 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="config-reloader" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714841 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="prom-label-proxy" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714849 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="alertmanager" Apr 23 17:48:11.714894 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.714854 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" containerName="kube-rbac-proxy-web" Apr 23 17:48:11.718228 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.718213 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.720319 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.720304 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 23 17:48:11.720434 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.720418 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9z6nr\"" Apr 23 17:48:11.720625 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.720605 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 23 17:48:11.720780 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.720631 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 23 17:48:11.720780 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.720684 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 23 17:48:11.720981 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.720964 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 23 17:48:11.721130 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.721089 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 23 17:48:11.721130 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.721125 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 23 17:48:11.721281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.721240 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 23 17:48:11.727069 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.727048 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 23 17:48:11.737581 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.737560 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:48:11.811572 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-web-config\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.811766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.811766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.811766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e838daf-b1b9-4d33-a33b-8cf40f382378-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.811766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e838daf-b1b9-4d33-a33b-8cf40f382378-config-out\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.811766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e838daf-b1b9-4d33-a33b-8cf40f382378-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.811766 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.812092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e838daf-b1b9-4d33-a33b-8cf40f382378-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.812092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82zc\" (UniqueName: \"kubernetes.io/projected/6e838daf-b1b9-4d33-a33b-8cf40f382378-kube-api-access-j82zc\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.812092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e838daf-b1b9-4d33-a33b-8cf40f382378-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.812092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.812092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.812092 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.811934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.912844 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.912747 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-web-config\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.912844 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.912799 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.912844 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.912827 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913097 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.912932 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e838daf-b1b9-4d33-a33b-8cf40f382378-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913097 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.912972 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e838daf-b1b9-4d33-a33b-8cf40f382378-config-out\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913097 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.912991 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e838daf-b1b9-4d33-a33b-8cf40f382378-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913097 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913042 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913097 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913072 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e838daf-b1b9-4d33-a33b-8cf40f382378-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913102 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j82zc\" (UniqueName: \"kubernetes.io/projected/6e838daf-b1b9-4d33-a33b-8cf40f382378-kube-api-access-j82zc\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913132 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e838daf-b1b9-4d33-a33b-8cf40f382378-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913156 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913193 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913356 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913222 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.913623 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.913445 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e838daf-b1b9-4d33-a33b-8cf40f382378-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.914175 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.914147 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e838daf-b1b9-4d33-a33b-8cf40f382378-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.915421 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.915351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e838daf-b1b9-4d33-a33b-8cf40f382378-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.915732 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.915707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e838daf-b1b9-4d33-a33b-8cf40f382378-config-out\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.915812 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.915792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.915957 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.915937 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.915999 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.915952 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.916290 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.916265 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.916673 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.916648 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.916804 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.916786 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e838daf-b1b9-4d33-a33b-8cf40f382378-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.916877 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.916854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-web-config\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.917764 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.917744 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e838daf-b1b9-4d33-a33b-8cf40f382378-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:11.932341 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:11.932323 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82zc\" (UniqueName: \"kubernetes.io/projected/6e838daf-b1b9-4d33-a33b-8cf40f382378-kube-api-access-j82zc\") pod \"alertmanager-main-0\" (UID: \"6e838daf-b1b9-4d33-a33b-8cf40f382378\") " pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:12.027648 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:12.027615 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 23 17:48:12.158519 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:12.158489 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 23 17:48:12.159633 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:48:12.159603 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e838daf_b1b9_4d33_a33b_8cf40f382378.slice/crio-ac408b622dd3334617682afa88347dfb2dd65d0c46207472fd23aa88e377206d WatchSource:0}: Error finding container ac408b622dd3334617682afa88347dfb2dd65d0c46207472fd23aa88e377206d: Status 404 returned error can't find the container with id ac408b622dd3334617682afa88347dfb2dd65d0c46207472fd23aa88e377206d Apr 23 17:48:12.657606 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:12.657553 2575 generic.go:358] "Generic (PLEG): container finished" podID="6e838daf-b1b9-4d33-a33b-8cf40f382378" containerID="e433cd5b90797db782412afc792920d889673ab82216edb95a01feb37daaa172" exitCode=0 Apr 23 17:48:12.657978 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:12.657632 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerDied","Data":"e433cd5b90797db782412afc792920d889673ab82216edb95a01feb37daaa172"} Apr 23 17:48:12.657978 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:12.657654 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"ac408b622dd3334617682afa88347dfb2dd65d0c46207472fd23aa88e377206d"} Apr 23 17:48:13.594268 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.594238 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64df7d2e-90c8-499a-b312-23d8393445cd" path="/var/lib/kubelet/pods/64df7d2e-90c8-499a-b312-23d8393445cd/volumes" Apr 23 17:48:13.663839 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.663806 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"cf3a54906123e73121e5506f02fd3a78b4362d66c26e9828bac82c3ec54f738c"} Apr 23 17:48:13.663839 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.663840 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"0df1866323dec7fd184db7ae0912e156dc7d3c1d4d8754c0de6c92db9400345c"} Apr 23 17:48:13.664233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.663851 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"f8295c8b7e6594ecb38d0fe578965845d21136871647033330ef695fe3e13f8f"} Apr 23 17:48:13.664233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.663860 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"d5288e388eda9cfade405a0b724304a9e3d604c5b4a6ebb1dc622da9a2b478f1"} Apr 23 17:48:13.664233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.663869 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"9ce80433dc710f24a5482b91c075175fb72ca53860a0ad27a37edcd6a0970e7e"} Apr 23 17:48:13.664233 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.663876 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e838daf-b1b9-4d33-a33b-8cf40f382378","Type":"ContainerStarted","Data":"1e4ec4c6472ce833dafd28156a93e008bb6f3104bce4646a09d1e80d7fb8ce96"} Apr 23 17:48:13.693289 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:13.693013 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.692991639 podStartE2EDuration="2.692991639s" podCreationTimestamp="2026-04-23 17:48:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:48:13.691872917 +0000 UTC m=+392.711325970" watchObservedRunningTime="2026-04-23 17:48:13.692991639 +0000 UTC m=+392.712444656" Apr 23 17:48:17.214502 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:17.214461 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:17.214502 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:17.214510 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:17.219391 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:17.219366 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:48:17.678851 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:48:17.678816 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:49:47.119195 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.119159 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng"] Apr 23 17:49:47.122156 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.122136 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.124332 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.124311 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Apr 23 17:49:47.124425 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.124379 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Apr 23 17:49:47.124875 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.124858 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Apr 23 17:49:47.125100 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.125085 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Apr 23 17:49:47.140047 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.140025 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng"] Apr 23 17:49:47.199367 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.199339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f97e9932-1d44-44a7-9a26-afe21dfd7d79-tmp\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.199489 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.199378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7q2\" (UniqueName: \"kubernetes.io/projected/f97e9932-1d44-44a7-9a26-afe21dfd7d79-kube-api-access-rt7q2\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.199489 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.199398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f97e9932-1d44-44a7-9a26-afe21dfd7d79-klusterlet-config\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.300355 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.300323 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f97e9932-1d44-44a7-9a26-afe21dfd7d79-tmp\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.300507 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.300370 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7q2\" (UniqueName: \"kubernetes.io/projected/f97e9932-1d44-44a7-9a26-afe21dfd7d79-kube-api-access-rt7q2\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.300507 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.300389 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f97e9932-1d44-44a7-9a26-afe21dfd7d79-klusterlet-config\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.300715 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.300694 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f97e9932-1d44-44a7-9a26-afe21dfd7d79-tmp\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.302716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.302698 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f97e9932-1d44-44a7-9a26-afe21dfd7d79-klusterlet-config\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.308206 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.308175 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7q2\" (UniqueName: \"kubernetes.io/projected/f97e9932-1d44-44a7-9a26-afe21dfd7d79-kube-api-access-rt7q2\") pod \"klusterlet-addon-workmgr-5856cd6b74-w64ng\" (UID: \"f97e9932-1d44-44a7-9a26-afe21dfd7d79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.430789 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.430721 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:47.548410 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.548379 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng"] Apr 23 17:49:47.550831 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:49:47.550804 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97e9932_1d44_44a7_9a26_afe21dfd7d79.slice/crio-2cc676599ae7eff751725631360fb08fc355d773d0cefb9fd8c1fcf34b1cce69 WatchSource:0}: Error finding container 2cc676599ae7eff751725631360fb08fc355d773d0cefb9fd8c1fcf34b1cce69: Status 404 returned error can't find the container with id 2cc676599ae7eff751725631360fb08fc355d773d0cefb9fd8c1fcf34b1cce69 Apr 23 17:49:47.909422 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:47.909391 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" event={"ID":"f97e9932-1d44-44a7-9a26-afe21dfd7d79","Type":"ContainerStarted","Data":"2cc676599ae7eff751725631360fb08fc355d773d0cefb9fd8c1fcf34b1cce69"} Apr 23 17:49:50.919655 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:50.919625 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" event={"ID":"f97e9932-1d44-44a7-9a26-afe21dfd7d79","Type":"ContainerStarted","Data":"a156aa02009c185a7004d83bac50aae29d58f580c6c4044167551368587d565e"} Apr 23 17:49:50.920006 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:50.919788 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:49:50.921275 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:50.921237 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" podUID="f97e9932-1d44-44a7-9a26-afe21dfd7d79" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.20:8000/readyz\": dial tcp 10.134.0.20:8000: connect: connection refused" Apr 23 17:49:50.937351 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:50.937167 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" podStartSLOduration=0.637525178 podStartE2EDuration="3.937151013s" podCreationTimestamp="2026-04-23 17:49:47 +0000 UTC" firstStartedPulling="2026-04-23 17:49:47.55240121 +0000 UTC m=+486.571854203" lastFinishedPulling="2026-04-23 17:49:50.852027028 +0000 UTC m=+489.871480038" observedRunningTime="2026-04-23 17:49:50.934842667 +0000 UTC m=+489.954295694" watchObservedRunningTime="2026-04-23 17:49:50.937151013 +0000 UTC m=+489.956604029" Apr 23 17:49:51.922970 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:49:51.922943 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5856cd6b74-w64ng" Apr 23 17:51:41.450133 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:51:41.450108 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:51:41.450624 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:51:41.450423 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:52:05.853252 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.853177 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b498c6d49-nlgtd"] Apr 23 17:52:05.856443 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.856428 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937414 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6bc\" (UniqueName: \"kubernetes.io/projected/6fcd778e-9bf3-428c-8028-600482b5ef44-kube-api-access-jf6bc\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937551 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fcd778e-9bf3-428c-8028-600482b5ef44-console-oauth-config\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937551 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fcd778e-9bf3-428c-8028-600482b5ef44-console-serving-cert\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937693 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-service-ca\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937693 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-oauth-serving-cert\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937693 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937665 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-console-config\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.937805 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.937694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-trusted-ca-bundle\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:05.962839 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:05.962816 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b498c6d49-nlgtd"] Apr 23 17:52:06.038522 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038488 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6bc\" (UniqueName: \"kubernetes.io/projected/6fcd778e-9bf3-428c-8028-600482b5ef44-kube-api-access-jf6bc\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.038716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038531 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fcd778e-9bf3-428c-8028-600482b5ef44-console-oauth-config\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.038716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038557 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fcd778e-9bf3-428c-8028-600482b5ef44-console-serving-cert\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.038716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038610 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-service-ca\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.038716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038642 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-oauth-serving-cert\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.038716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038672 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-console-config\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.038716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.038696 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-trusted-ca-bundle\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.039404 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.039378 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-service-ca\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.039520 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.039491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-oauth-serving-cert\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.039520 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.039508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-trusted-ca-bundle\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.039620 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.039562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fcd778e-9bf3-428c-8028-600482b5ef44-console-config\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.040858 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.040835 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fcd778e-9bf3-428c-8028-600482b5ef44-console-oauth-config\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.041131 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.041112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fcd778e-9bf3-428c-8028-600482b5ef44-console-serving-cert\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.047391 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.047362 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6bc\" (UniqueName: \"kubernetes.io/projected/6fcd778e-9bf3-428c-8028-600482b5ef44-kube-api-access-jf6bc\") pod \"console-5b498c6d49-nlgtd\" (UID: \"6fcd778e-9bf3-428c-8028-600482b5ef44\") " pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.164687 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.164583 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:06.283472 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.283442 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b498c6d49-nlgtd"] Apr 23 17:52:06.287246 ip-10-0-137-68 kubenswrapper[2575]: W0423 17:52:06.287221 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcd778e_9bf3_428c_8028_600482b5ef44.slice/crio-8e52038a0f7dd7f531396f293ef3f2784cf8fc3ce6c2e9de2c8c05e483cf5f4d WatchSource:0}: Error finding container 8e52038a0f7dd7f531396f293ef3f2784cf8fc3ce6c2e9de2c8c05e483cf5f4d: Status 404 returned error can't find the container with id 8e52038a0f7dd7f531396f293ef3f2784cf8fc3ce6c2e9de2c8c05e483cf5f4d Apr 23 17:52:06.288838 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:06.288805 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 17:52:07.278328 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:07.278294 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b498c6d49-nlgtd" event={"ID":"6fcd778e-9bf3-428c-8028-600482b5ef44","Type":"ContainerStarted","Data":"86c29cd675782d6668190f91e2213b593b2fa7096e6bc251acb30f3271dee46d"} Apr 23 17:52:07.278328 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:07.278331 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b498c6d49-nlgtd" event={"ID":"6fcd778e-9bf3-428c-8028-600482b5ef44","Type":"ContainerStarted","Data":"8e52038a0f7dd7f531396f293ef3f2784cf8fc3ce6c2e9de2c8c05e483cf5f4d"} Apr 23 17:52:07.299927 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:07.299877 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5b498c6d49-nlgtd" podStartSLOduration=2.299864369 podStartE2EDuration="2.299864369s" podCreationTimestamp="2026-04-23 17:52:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 17:52:07.298013923 +0000 UTC m=+626.317466937" watchObservedRunningTime="2026-04-23 17:52:07.299864369 +0000 UTC m=+626.319317384" Apr 23 17:52:16.164974 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:16.164936 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:16.165385 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:16.165026 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:16.170337 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:16.170313 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:16.306544 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:16.306515 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b498c6d49-nlgtd" Apr 23 17:52:16.356802 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:16.356771 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c54c7699-h72c2"] Apr 23 17:52:41.376339 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.376284 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-69c54c7699-h72c2" podUID="91a79641-ed93-4060-82ee-6a919793a7a4" containerName="console" containerID="cri-o://97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268" gracePeriod=15 Apr 23 17:52:41.604267 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.604246 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69c54c7699-h72c2_91a79641-ed93-4060-82ee-6a919793a7a4/console/0.log" Apr 23 17:52:41.604381 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.604302 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:52:41.719716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719624 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-serving-cert\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.719716 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719675 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrnjj\" (UniqueName: \"kubernetes.io/projected/91a79641-ed93-4060-82ee-6a919793a7a4-kube-api-access-jrnjj\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.719916 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719725 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-service-ca\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.719916 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719754 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-trusted-ca-bundle\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.719916 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719796 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-console-config\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.719916 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719844 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-oauth-serving-cert\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.719916 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.719867 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-oauth-config\") pod \"91a79641-ed93-4060-82ee-6a919793a7a4\" (UID: \"91a79641-ed93-4060-82ee-6a919793a7a4\") " Apr 23 17:52:41.720201 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.720171 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:52:41.720470 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.720175 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-console-config" (OuterVolumeSpecName: "console-config") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:52:41.720470 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.720195 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:52:41.720470 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.720383 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 23 17:52:41.721955 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.721935 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:52:41.722017 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.721981 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 17:52:41.722017 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.722000 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a79641-ed93-4060-82ee-6a919793a7a4-kube-api-access-jrnjj" (OuterVolumeSpecName: "kube-api-access-jrnjj") pod "91a79641-ed93-4060-82ee-6a919793a7a4" (UID: "91a79641-ed93-4060-82ee-6a919793a7a4"). InnerVolumeSpecName "kube-api-access-jrnjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 17:52:41.821095 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821052 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-serving-cert\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:41.821095 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821090 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnjj\" (UniqueName: \"kubernetes.io/projected/91a79641-ed93-4060-82ee-6a919793a7a4-kube-api-access-jrnjj\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:41.821095 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821100 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-service-ca\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:41.821095 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821108 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-trusted-ca-bundle\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:41.821393 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821117 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-console-config\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:41.821393 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821125 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a79641-ed93-4060-82ee-6a919793a7a4-oauth-serving-cert\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:41.821393 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:41.821133 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a79641-ed93-4060-82ee-6a919793a7a4-console-oauth-config\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 17:52:42.373119 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.373091 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69c54c7699-h72c2_91a79641-ed93-4060-82ee-6a919793a7a4/console/0.log" Apr 23 17:52:42.373281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.373129 2575 generic.go:358] "Generic (PLEG): container finished" podID="91a79641-ed93-4060-82ee-6a919793a7a4" containerID="97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268" exitCode=2 Apr 23 17:52:42.373281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.373198 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c54c7699-h72c2" event={"ID":"91a79641-ed93-4060-82ee-6a919793a7a4","Type":"ContainerDied","Data":"97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268"} Apr 23 17:52:42.373281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.373199 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c54c7699-h72c2" Apr 23 17:52:42.373281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.373224 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c54c7699-h72c2" event={"ID":"91a79641-ed93-4060-82ee-6a919793a7a4","Type":"ContainerDied","Data":"f1c1eb57e94427e11f522259cff896f07c01ad9e1b6af7e74fe863f157f2a8a2"} Apr 23 17:52:42.373281 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.373238 2575 scope.go:117] "RemoveContainer" containerID="97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268" Apr 23 17:52:42.381292 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.381113 2575 scope.go:117] "RemoveContainer" containerID="97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268" Apr 23 17:52:42.381531 ip-10-0-137-68 kubenswrapper[2575]: E0423 17:52:42.381354 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268\": container with ID starting with 97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268 not found: ID does not exist" containerID="97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268" Apr 23 17:52:42.381531 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.381389 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268"} err="failed to get container status \"97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268\": rpc error: code = NotFound desc = could not find container \"97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268\": container with ID starting with 97b713103f1cedc526a9b39f5c87ba5f4b1bbe2f669091fe6e04efcb4181f268 not found: ID does not exist" Apr 23 17:52:42.392915 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.392887 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c54c7699-h72c2"] Apr 23 17:52:42.396807 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:42.396788 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69c54c7699-h72c2"] Apr 23 17:52:43.593970 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:52:43.593937 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a79641-ed93-4060-82ee-6a919793a7a4" path="/var/lib/kubelet/pods/91a79641-ed93-4060-82ee-6a919793a7a4/volumes" Apr 23 17:56:41.468831 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:56:41.468800 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 17:56:41.469733 ip-10-0-137-68 kubenswrapper[2575]: I0423 17:56:41.469709 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:01:41.491453 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:01:41.491426 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:01:41.495304 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:01:41.495282 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:06:41.511924 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:06:41.511896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:06:41.514554 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:06:41.514535 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:11:41.535931 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:11:41.535809 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:11:41.542581 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:11:41.538801 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:15:32.352317 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.352277 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-rlg54"] Apr 23 18:15:32.352838 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.352569 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a79641-ed93-4060-82ee-6a919793a7a4" containerName="console" Apr 23 18:15:32.352838 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.352581 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a79641-ed93-4060-82ee-6a919793a7a4" containerName="console" Apr 23 18:15:32.352838 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.352649 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="91a79641-ed93-4060-82ee-6a919793a7a4" containerName="console" Apr 23 18:15:32.355387 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.355367 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.357426 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.357389 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Apr 23 18:15:32.358367 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.358347 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Apr 23 18:15:32.358476 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.358382 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Apr 23 18:15:32.359048 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.359028 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Apr 23 18:15:32.359128 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.359067 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Apr 23 18:15:32.359128 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.359080 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6clmg\"" Apr 23 18:15:32.367533 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.367507 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-rlg54"] Apr 23 18:15:32.384581 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.384546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/933f1f96-f06c-451e-a00c-f893db98c39f-cabundle0\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.384807 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.384639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8mk\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-kube-api-access-xv8mk\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.384807 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.384682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.485223 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.485177 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.485439 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.485262 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/933f1f96-f06c-451e-a00c-f893db98c39f-cabundle0\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.485439 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.485297 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8mk\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-kube-api-access-xv8mk\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.485439 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.485325 2575 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Apr 23 18:15:32.485439 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.485353 2575 secret.go:281] references non-existent secret key: ca.crt Apr 23 18:15:32.485439 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.485364 2575 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 23 18:15:32.485439 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.485379 2575 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-rlg54: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Apr 23 18:15:32.485783 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.485446 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates podName:933f1f96-f06c-451e-a00c-f893db98c39f nodeName:}" failed. No retries permitted until 2026-04-23 18:15:32.985427115 +0000 UTC m=+2032.004880114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates") pod "keda-operator-ffbb595cb-rlg54" (UID: "933f1f96-f06c-451e-a00c-f893db98c39f") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Apr 23 18:15:32.485990 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.485970 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/933f1f96-f06c-451e-a00c-f893db98c39f-cabundle0\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.495262 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.495223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8mk\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-kube-api-access-xv8mk\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.990180 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:32.990139 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:32.990397 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.990299 2575 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Apr 23 18:15:32.990397 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.990327 2575 secret.go:281] references non-existent secret key: ca.crt Apr 23 18:15:32.990397 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.990338 2575 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 23 18:15:32.990397 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.990354 2575 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-rlg54: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Apr 23 18:15:32.990620 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:32.990437 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates podName:933f1f96-f06c-451e-a00c-f893db98c39f nodeName:}" failed. No retries permitted until 2026-04-23 18:15:33.990414348 +0000 UTC m=+2033.009867345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates") pod "keda-operator-ffbb595cb-rlg54" (UID: "933f1f96-f06c-451e-a00c-f893db98c39f") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Apr 23 18:15:33.995541 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:33.995481 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:33.996082 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:33.995680 2575 secret.go:281] references non-existent secret key: ca.crt Apr 23 18:15:33.996082 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:33.995707 2575 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 23 18:15:33.996082 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:33.995723 2575 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-rlg54: references non-existent secret key: ca.crt Apr 23 18:15:33.996082 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:15:33.995798 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates podName:933f1f96-f06c-451e-a00c-f893db98c39f nodeName:}" failed. No retries permitted until 2026-04-23 18:15:35.995776883 +0000 UTC m=+2035.015229877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates") pod "keda-operator-ffbb595cb-rlg54" (UID: "933f1f96-f06c-451e-a00c-f893db98c39f") : references non-existent secret key: ca.crt Apr 23 18:15:36.010906 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:36.010860 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:36.013387 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:36.013357 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/933f1f96-f06c-451e-a00c-f893db98c39f-certificates\") pod \"keda-operator-ffbb595cb-rlg54\" (UID: \"933f1f96-f06c-451e-a00c-f893db98c39f\") " pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:36.266313 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:36.266208 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:15:36.391885 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:36.391709 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-rlg54"] Apr 23 18:15:36.394704 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:15:36.394672 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933f1f96_f06c_451e_a00c_f893db98c39f.slice/crio-94f66c00b0dd365d169584fa0c621c4f500b6e2a88374fc3547fdc6c9697c275 WatchSource:0}: Error finding container 94f66c00b0dd365d169584fa0c621c4f500b6e2a88374fc3547fdc6c9697c275: Status 404 returned error can't find the container with id 94f66c00b0dd365d169584fa0c621c4f500b6e2a88374fc3547fdc6c9697c275 Apr 23 18:15:36.395884 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:36.395862 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 18:15:37.109497 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:37.109455 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" event={"ID":"933f1f96-f06c-451e-a00c-f893db98c39f","Type":"ContainerStarted","Data":"94f66c00b0dd365d169584fa0c621c4f500b6e2a88374fc3547fdc6c9697c275"} Apr 23 18:15:41.124262 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:41.124221 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" event={"ID":"933f1f96-f06c-451e-a00c-f893db98c39f","Type":"ContainerStarted","Data":"033640c8b70ae38927fc4f320d6df9c20e0ca9e38b21cd511d8380c1fb606314"} Apr 23 18:15:41.124707 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:15:41.124374 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:16:02.129868 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:02.129796 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" Apr 23 18:16:02.149864 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:02.149804 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-rlg54" podStartSLOduration=25.976027631 podStartE2EDuration="30.149791787s" podCreationTimestamp="2026-04-23 18:15:32 +0000 UTC" firstStartedPulling="2026-04-23 18:15:36.395992336 +0000 UTC m=+2035.415445330" lastFinishedPulling="2026-04-23 18:15:40.569756492 +0000 UTC m=+2039.589209486" observedRunningTime="2026-04-23 18:15:41.149582546 +0000 UTC m=+2040.169035565" watchObservedRunningTime="2026-04-23 18:16:02.149791787 +0000 UTC m=+2061.169244803" Apr 23 18:16:38.794457 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.794422 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw"] Apr 23 18:16:38.797530 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.797511 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:38.799804 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.799771 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ckc8r\"" Apr 23 18:16:38.799907 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.799872 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Apr 23 18:16:38.799986 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.799964 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Apr 23 18:16:38.810900 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.810870 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw"] Apr 23 18:16:38.889997 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.889955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6cv\" (UniqueName: \"kubernetes.io/projected/48b84612-e4a8-4b2d-8008-e43bb73b1ac6-kube-api-access-rh6cv\") pod \"cert-manager-operator-controller-manager-54b9655956-sd7dw\" (UID: \"48b84612-e4a8-4b2d-8008-e43bb73b1ac6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:38.890175 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.890005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b84612-e4a8-4b2d-8008-e43bb73b1ac6-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-sd7dw\" (UID: \"48b84612-e4a8-4b2d-8008-e43bb73b1ac6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:38.991458 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.991414 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6cv\" (UniqueName: \"kubernetes.io/projected/48b84612-e4a8-4b2d-8008-e43bb73b1ac6-kube-api-access-rh6cv\") pod \"cert-manager-operator-controller-manager-54b9655956-sd7dw\" (UID: \"48b84612-e4a8-4b2d-8008-e43bb73b1ac6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:38.991458 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.991462 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b84612-e4a8-4b2d-8008-e43bb73b1ac6-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-sd7dw\" (UID: \"48b84612-e4a8-4b2d-8008-e43bb73b1ac6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:38.991833 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:38.991817 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b84612-e4a8-4b2d-8008-e43bb73b1ac6-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-sd7dw\" (UID: \"48b84612-e4a8-4b2d-8008-e43bb73b1ac6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:39.000951 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:39.000923 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6cv\" (UniqueName: \"kubernetes.io/projected/48b84612-e4a8-4b2d-8008-e43bb73b1ac6-kube-api-access-rh6cv\") pod \"cert-manager-operator-controller-manager-54b9655956-sd7dw\" (UID: \"48b84612-e4a8-4b2d-8008-e43bb73b1ac6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:39.106487 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:39.106394 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" Apr 23 18:16:39.260333 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:39.260309 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw"] Apr 23 18:16:39.263092 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:16:39.263054 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b84612_e4a8_4b2d_8008_e43bb73b1ac6.slice/crio-797e86223883795d8e38c443e1cd29eb73b3076f68b66e8d8cde08c16201223c WatchSource:0}: Error finding container 797e86223883795d8e38c443e1cd29eb73b3076f68b66e8d8cde08c16201223c: Status 404 returned error can't find the container with id 797e86223883795d8e38c443e1cd29eb73b3076f68b66e8d8cde08c16201223c Apr 23 18:16:39.284992 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:39.284960 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" event={"ID":"48b84612-e4a8-4b2d-8008-e43bb73b1ac6","Type":"ContainerStarted","Data":"797e86223883795d8e38c443e1cd29eb73b3076f68b66e8d8cde08c16201223c"} Apr 23 18:16:41.556320 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:41.556201 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:16:41.567700 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:41.557651 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:16:42.298177 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:42.298149 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" event={"ID":"48b84612-e4a8-4b2d-8008-e43bb73b1ac6","Type":"ContainerStarted","Data":"6e04b50004e21e972354ba2dc3b2348a7ed01e7dd955a7f7cb45b207895aee45"} Apr 23 18:16:42.321401 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:42.321349 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-sd7dw" podStartSLOduration=1.9845900250000001 podStartE2EDuration="4.321333591s" podCreationTimestamp="2026-04-23 18:16:38 +0000 UTC" firstStartedPulling="2026-04-23 18:16:39.265660976 +0000 UTC m=+2098.285113984" lastFinishedPulling="2026-04-23 18:16:41.602404556 +0000 UTC m=+2100.621857550" observedRunningTime="2026-04-23 18:16:42.319819747 +0000 UTC m=+2101.339272786" watchObservedRunningTime="2026-04-23 18:16:42.321333591 +0000 UTC m=+2101.340786607" Apr 23 18:16:44.925137 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:44.925098 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-d8986"] Apr 23 18:16:44.928342 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:44.928322 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:44.930716 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:44.930695 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-529s4\"" Apr 23 18:16:44.930903 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:44.930884 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Apr 23 18:16:44.931368 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:44.931350 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Apr 23 18:16:44.939697 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:44.939673 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-d8986"] Apr 23 18:16:45.041353 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.041316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbe14f1e-5956-455c-b7c7-e47ae0dd0826-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-d8986\" (UID: \"dbe14f1e-5956-455c-b7c7-e47ae0dd0826\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.041539 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.041361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2ls\" (UniqueName: \"kubernetes.io/projected/dbe14f1e-5956-455c-b7c7-e47ae0dd0826-kube-api-access-lz2ls\") pod \"cert-manager-webhook-587ccfb98-d8986\" (UID: \"dbe14f1e-5956-455c-b7c7-e47ae0dd0826\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.141844 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.141806 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2ls\" (UniqueName: \"kubernetes.io/projected/dbe14f1e-5956-455c-b7c7-e47ae0dd0826-kube-api-access-lz2ls\") pod \"cert-manager-webhook-587ccfb98-d8986\" (UID: \"dbe14f1e-5956-455c-b7c7-e47ae0dd0826\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.142059 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.141913 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbe14f1e-5956-455c-b7c7-e47ae0dd0826-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-d8986\" (UID: \"dbe14f1e-5956-455c-b7c7-e47ae0dd0826\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.150437 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.150399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbe14f1e-5956-455c-b7c7-e47ae0dd0826-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-d8986\" (UID: \"dbe14f1e-5956-455c-b7c7-e47ae0dd0826\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.150554 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.150491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2ls\" (UniqueName: \"kubernetes.io/projected/dbe14f1e-5956-455c-b7c7-e47ae0dd0826-kube-api-access-lz2ls\") pod \"cert-manager-webhook-587ccfb98-d8986\" (UID: \"dbe14f1e-5956-455c-b7c7-e47ae0dd0826\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.244417 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.244381 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:45.366264 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:45.366232 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-d8986"] Apr 23 18:16:45.369458 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:16:45.369423 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbe14f1e_5956_455c_b7c7_e47ae0dd0826.slice/crio-5890a0014db35920abc0420ce1655d86556b197219f629dc788852fc5302b8d1 WatchSource:0}: Error finding container 5890a0014db35920abc0420ce1655d86556b197219f629dc788852fc5302b8d1: Status 404 returned error can't find the container with id 5890a0014db35920abc0420ce1655d86556b197219f629dc788852fc5302b8d1 Apr 23 18:16:46.117996 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.117948 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-s6bk2"] Apr 23 18:16:46.121405 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.121377 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.123937 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.123911 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k78mn\"" Apr 23 18:16:46.128303 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.128277 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-s6bk2"] Apr 23 18:16:46.251364 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.251322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f38373-497b-4612-b03b-7dc84fc3b441-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-s6bk2\" (UID: \"d0f38373-497b-4612-b03b-7dc84fc3b441\") " pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.251556 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.251372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zljs\" (UniqueName: \"kubernetes.io/projected/d0f38373-497b-4612-b03b-7dc84fc3b441-kube-api-access-5zljs\") pod \"cert-manager-cainjector-68b757865b-s6bk2\" (UID: \"d0f38373-497b-4612-b03b-7dc84fc3b441\") " pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.312697 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.312659 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" event={"ID":"dbe14f1e-5956-455c-b7c7-e47ae0dd0826","Type":"ContainerStarted","Data":"5890a0014db35920abc0420ce1655d86556b197219f629dc788852fc5302b8d1"} Apr 23 18:16:46.352422 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.352386 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f38373-497b-4612-b03b-7dc84fc3b441-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-s6bk2\" (UID: \"d0f38373-497b-4612-b03b-7dc84fc3b441\") " pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.352628 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.352428 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5zljs\" (UniqueName: \"kubernetes.io/projected/d0f38373-497b-4612-b03b-7dc84fc3b441-kube-api-access-5zljs\") pod \"cert-manager-cainjector-68b757865b-s6bk2\" (UID: \"d0f38373-497b-4612-b03b-7dc84fc3b441\") " pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.360663 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.360632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f38373-497b-4612-b03b-7dc84fc3b441-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-s6bk2\" (UID: \"d0f38373-497b-4612-b03b-7dc84fc3b441\") " pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.360850 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.360815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zljs\" (UniqueName: \"kubernetes.io/projected/d0f38373-497b-4612-b03b-7dc84fc3b441-kube-api-access-5zljs\") pod \"cert-manager-cainjector-68b757865b-s6bk2\" (UID: \"d0f38373-497b-4612-b03b-7dc84fc3b441\") " pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.434560 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.434461 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" Apr 23 18:16:46.585195 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:46.585015 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-s6bk2"] Apr 23 18:16:46.588198 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:16:46.588168 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0f38373_497b_4612_b03b_7dc84fc3b441.slice/crio-404eb59ec93cbb597256bcdc36309f763931ca4269b9ea387f84902ce663181a WatchSource:0}: Error finding container 404eb59ec93cbb597256bcdc36309f763931ca4269b9ea387f84902ce663181a: Status 404 returned error can't find the container with id 404eb59ec93cbb597256bcdc36309f763931ca4269b9ea387f84902ce663181a Apr 23 18:16:47.318194 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:47.318152 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" event={"ID":"d0f38373-497b-4612-b03b-7dc84fc3b441","Type":"ContainerStarted","Data":"404eb59ec93cbb597256bcdc36309f763931ca4269b9ea387f84902ce663181a"} Apr 23 18:16:48.323583 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:48.323542 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" event={"ID":"d0f38373-497b-4612-b03b-7dc84fc3b441","Type":"ContainerStarted","Data":"3a731a1bed32056d54ce6390ec48fa52956b0228ba5a699abe8d01a1d83fbfc3"} Apr 23 18:16:48.325128 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:48.325098 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" event={"ID":"dbe14f1e-5956-455c-b7c7-e47ae0dd0826","Type":"ContainerStarted","Data":"b010dbbbf656ad8376624a485dea9bd910dbe96a6c74b99e22ff51a5ab72fff4"} Apr 23 18:16:48.325247 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:48.325222 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:16:48.343099 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:48.343040 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-s6bk2" podStartSLOduration=0.811232605 podStartE2EDuration="2.343019501s" podCreationTimestamp="2026-04-23 18:16:46 +0000 UTC" firstStartedPulling="2026-04-23 18:16:46.590443278 +0000 UTC m=+2105.609896272" lastFinishedPulling="2026-04-23 18:16:48.122230171 +0000 UTC m=+2107.141683168" observedRunningTime="2026-04-23 18:16:48.341734293 +0000 UTC m=+2107.361187309" watchObservedRunningTime="2026-04-23 18:16:48.343019501 +0000 UTC m=+2107.362472519" Apr 23 18:16:48.360883 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:48.360764 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" podStartSLOduration=1.616174743 podStartE2EDuration="4.360747707s" podCreationTimestamp="2026-04-23 18:16:44 +0000 UTC" firstStartedPulling="2026-04-23 18:16:45.371481309 +0000 UTC m=+2104.390934303" lastFinishedPulling="2026-04-23 18:16:48.116054271 +0000 UTC m=+2107.135507267" observedRunningTime="2026-04-23 18:16:48.359258855 +0000 UTC m=+2107.378711872" watchObservedRunningTime="2026-04-23 18:16:48.360747707 +0000 UTC m=+2107.380200723" Apr 23 18:16:54.331407 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:16:54.331372 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-d8986" Apr 23 18:17:02.332219 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.332180 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx"] Apr 23 18:17:02.338757 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.338736 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.340988 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.340954 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6qw6d\"" Apr 23 18:17:02.340988 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.340982 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Apr 23 18:17:02.341307 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.341287 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Apr 23 18:17:02.344025 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.343999 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx"] Apr 23 18:17:02.489018 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.488964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jvsc\" (UniqueName: \"kubernetes.io/projected/eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46-kube-api-access-7jvsc\") pod \"openshift-lws-operator-bfc7f696d-wxdcx\" (UID: \"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.489224 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.489085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46-tmp\") pod \"openshift-lws-operator-bfc7f696d-wxdcx\" (UID: \"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.590258 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.590157 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46-tmp\") pod \"openshift-lws-operator-bfc7f696d-wxdcx\" (UID: \"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.590258 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.590205 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7jvsc\" (UniqueName: \"kubernetes.io/projected/eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46-kube-api-access-7jvsc\") pod \"openshift-lws-operator-bfc7f696d-wxdcx\" (UID: \"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.590632 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.590608 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46-tmp\") pod \"openshift-lws-operator-bfc7f696d-wxdcx\" (UID: \"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.598600 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.598557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jvsc\" (UniqueName: \"kubernetes.io/projected/eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46-kube-api-access-7jvsc\") pod \"openshift-lws-operator-bfc7f696d-wxdcx\" (UID: \"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.649292 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.649247 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" Apr 23 18:17:02.779926 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:02.779837 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx"] Apr 23 18:17:02.782604 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:17:02.782555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb4957a3_2ef0_4ac0_ba04_2a568e4a4f46.slice/crio-4d8a78160046cda571f57b42d0240710b2fbaee7bdac7a0c1adc0dcb4668ba6b WatchSource:0}: Error finding container 4d8a78160046cda571f57b42d0240710b2fbaee7bdac7a0c1adc0dcb4668ba6b: Status 404 returned error can't find the container with id 4d8a78160046cda571f57b42d0240710b2fbaee7bdac7a0c1adc0dcb4668ba6b Apr 23 18:17:03.372073 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:03.372040 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" event={"ID":"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46","Type":"ContainerStarted","Data":"4d8a78160046cda571f57b42d0240710b2fbaee7bdac7a0c1adc0dcb4668ba6b"} Apr 23 18:17:06.384370 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:06.384331 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" event={"ID":"eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46","Type":"ContainerStarted","Data":"25285ae922ed71189600ef171f8bea304a6ef5769ade9667219d608d7a90e52e"} Apr 23 18:17:06.403015 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:06.402961 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-wxdcx" podStartSLOduration=1.481570375 podStartE2EDuration="4.402947363s" podCreationTimestamp="2026-04-23 18:17:02 +0000 UTC" firstStartedPulling="2026-04-23 18:17:02.784075426 +0000 UTC m=+2121.803528433" lastFinishedPulling="2026-04-23 18:17:05.705452424 +0000 UTC m=+2124.724905421" observedRunningTime="2026-04-23 18:17:06.402459746 +0000 UTC m=+2125.421912763" watchObservedRunningTime="2026-04-23 18:17:06.402947363 +0000 UTC m=+2125.422400378" Apr 23 18:17:23.270370 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.270333 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m"] Apr 23 18:17:23.273629 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.273607 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.276501 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.276479 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Apr 23 18:17:23.276501 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.276495 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xlk9l\"" Apr 23 18:17:23.276695 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.276488 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Apr 23 18:17:23.276695 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.276484 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Apr 23 18:17:23.283471 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.283449 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m"] Apr 23 18:17:23.362455 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.362418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzht9\" (UniqueName: \"kubernetes.io/projected/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-kube-api-access-dzht9\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.362455 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.362460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-manager-config\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.362763 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.362555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-metrics-cert\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.362763 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.362639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-cert\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.463791 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.463747 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dzht9\" (UniqueName: \"kubernetes.io/projected/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-kube-api-access-dzht9\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.463791 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.463795 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-manager-config\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.464018 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.463831 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-metrics-cert\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.464018 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.463864 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-cert\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.464528 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.464500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-manager-config\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.466273 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.466251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-metrics-cert\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.466460 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.466439 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-cert\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.473044 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.473015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzht9\" (UniqueName: \"kubernetes.io/projected/b132adb9-73b5-43bd-b408-9d6da8e3ba4a-kube-api-access-dzht9\") pod \"lws-controller-manager-7bc7774745-kjk2m\" (UID: \"b132adb9-73b5-43bd-b408-9d6da8e3ba4a\") " pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.583818 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.583721 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:23.710784 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:23.710758 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m"] Apr 23 18:17:23.713509 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:17:23.713476 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb132adb9_73b5_43bd_b408_9d6da8e3ba4a.slice/crio-f321045d792cc40301fac0f1f792b137fd9fb32e2860707b631330a5ee2a0c79 WatchSource:0}: Error finding container f321045d792cc40301fac0f1f792b137fd9fb32e2860707b631330a5ee2a0c79: Status 404 returned error can't find the container with id f321045d792cc40301fac0f1f792b137fd9fb32e2860707b631330a5ee2a0c79 Apr 23 18:17:24.445367 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:24.445258 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" event={"ID":"b132adb9-73b5-43bd-b408-9d6da8e3ba4a","Type":"ContainerStarted","Data":"f321045d792cc40301fac0f1f792b137fd9fb32e2860707b631330a5ee2a0c79"} Apr 23 18:17:25.449425 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:25.449329 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" event={"ID":"b132adb9-73b5-43bd-b408-9d6da8e3ba4a","Type":"ContainerStarted","Data":"fb2111d4edf30529edcc54c8fb9bd50a98f21c3846d5b2334cf8fc132a075704"} Apr 23 18:17:25.449873 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:25.449469 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:17:25.467374 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:25.467317 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" podStartSLOduration=1.027373002 podStartE2EDuration="2.46730119s" podCreationTimestamp="2026-04-23 18:17:23 +0000 UTC" firstStartedPulling="2026-04-23 18:17:23.715416978 +0000 UTC m=+2142.734869972" lastFinishedPulling="2026-04-23 18:17:25.155345163 +0000 UTC m=+2144.174798160" observedRunningTime="2026-04-23 18:17:25.466204279 +0000 UTC m=+2144.485657293" watchObservedRunningTime="2026-04-23 18:17:25.46730119 +0000 UTC m=+2144.486754206" Apr 23 18:17:36.455076 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:17:36.455040 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bc7774745-kjk2m" Apr 23 18:18:15.540929 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.540895 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv"] Apr 23 18:18:15.543999 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.543983 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:18:15.546217 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.546192 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Apr 23 18:18:15.546435 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.546418 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Apr 23 18:18:15.546489 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.546456 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Apr 23 18:18:15.546536 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.546491 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pg4sw\"" Apr 23 18:18:15.554866 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.554843 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv"] Apr 23 18:18:15.605767 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.605719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6js9\" (UniqueName: \"kubernetes.io/projected/a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b-kube-api-access-h6js9\") pod \"dns-operator-controller-manager-844548ff4c-hhjsv\" (UID: \"a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:18:15.706573 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.706530 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h6js9\" (UniqueName: \"kubernetes.io/projected/a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b-kube-api-access-h6js9\") pod \"dns-operator-controller-manager-844548ff4c-hhjsv\" (UID: \"a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:18:15.720398 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.720359 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6js9\" (UniqueName: \"kubernetes.io/projected/a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b-kube-api-access-h6js9\") pod \"dns-operator-controller-manager-844548ff4c-hhjsv\" (UID: \"a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:18:15.855255 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.855161 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:18:15.988294 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:15.988257 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv"] Apr 23 18:18:15.991787 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:18:15.991757 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46f9ddd_b4c3_45e1_9887_9e9fd258cd6b.slice/crio-528f28f2c1b0f322d76016286f162a11683cb7ff0aa40896ae8e69d2c5d2f6c8 WatchSource:0}: Error finding container 528f28f2c1b0f322d76016286f162a11683cb7ff0aa40896ae8e69d2c5d2f6c8: Status 404 returned error can't find the container with id 528f28f2c1b0f322d76016286f162a11683cb7ff0aa40896ae8e69d2c5d2f6c8 Apr 23 18:18:16.622309 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:16.622265 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" event={"ID":"a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b","Type":"ContainerStarted","Data":"528f28f2c1b0f322d76016286f162a11683cb7ff0aa40896ae8e69d2c5d2f6c8"} Apr 23 18:18:17.686287 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.686192 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9"] Apr 23 18:18:17.697446 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.697404 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9"] Apr 23 18:18:17.697657 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.697538 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.699720 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.699507 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Apr 23 18:18:17.699720 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.699561 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m4rdk\"" Apr 23 18:18:17.699934 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.699874 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Apr 23 18:18:17.724126 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.724087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c0f2c4-353c-4741-81a8-57f026b023d9-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.724327 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.724234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd96p\" (UniqueName: \"kubernetes.io/projected/a0c0f2c4-353c-4741-81a8-57f026b023d9-kube-api-access-bd96p\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.724327 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.724279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0c0f2c4-353c-4741-81a8-57f026b023d9-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.825205 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.825154 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bd96p\" (UniqueName: \"kubernetes.io/projected/a0c0f2c4-353c-4741-81a8-57f026b023d9-kube-api-access-bd96p\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.825512 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.825217 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0c0f2c4-353c-4741-81a8-57f026b023d9-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.825512 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.825294 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c0f2c4-353c-4741-81a8-57f026b023d9-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.825512 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:18:17.825449 2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Apr 23 18:18:17.825732 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:18:17.825527 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c0f2c4-353c-4741-81a8-57f026b023d9-plugin-serving-cert podName:a0c0f2c4-353c-4741-81a8-57f026b023d9 nodeName:}" failed. No retries permitted until 2026-04-23 18:18:18.325505496 +0000 UTC m=+2197.344958497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a0c0f2c4-353c-4741-81a8-57f026b023d9-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-mj4k9" (UID: "a0c0f2c4-353c-4741-81a8-57f026b023d9") : secret "plugin-serving-cert" not found Apr 23 18:18:17.825966 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.825942 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0c0f2c4-353c-4741-81a8-57f026b023d9-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:17.835909 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:17.835877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd96p\" (UniqueName: \"kubernetes.io/projected/a0c0f2c4-353c-4741-81a8-57f026b023d9-kube-api-access-bd96p\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:18.330103 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.330068 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c0f2c4-353c-4741-81a8-57f026b023d9-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:18.332513 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.332479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c0f2c4-353c-4741-81a8-57f026b023d9-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-mj4k9\" (UID: \"a0c0f2c4-353c-4741-81a8-57f026b023d9\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:18.611552 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.611497 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" Apr 23 18:18:18.630777 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.630734 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" event={"ID":"a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b","Type":"ContainerStarted","Data":"809d0bed89f47c46e85aa51d7d6607970f7ec8abf371010f8d2cff1f7b0ad520"} Apr 23 18:18:18.630951 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.630889 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:18:18.652731 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.652663 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" podStartSLOduration=1.425164042 podStartE2EDuration="3.65263887s" podCreationTimestamp="2026-04-23 18:18:15 +0000 UTC" firstStartedPulling="2026-04-23 18:18:15.993764337 +0000 UTC m=+2195.013217332" lastFinishedPulling="2026-04-23 18:18:18.221239163 +0000 UTC m=+2197.240692160" observedRunningTime="2026-04-23 18:18:18.651099118 +0000 UTC m=+2197.670552136" watchObservedRunningTime="2026-04-23 18:18:18.65263887 +0000 UTC m=+2197.672091887" Apr 23 18:18:18.746066 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:18.746030 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9"] Apr 23 18:18:18.750328 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:18:18.750300 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0c0f2c4_353c_4741_81a8_57f026b023d9.slice/crio-45c3805baef083eb6a926562dbfea57d28d5d2f9dbf4f9b1a4bc58f8a1064f31 WatchSource:0}: Error finding container 45c3805baef083eb6a926562dbfea57d28d5d2f9dbf4f9b1a4bc58f8a1064f31: Status 404 returned error can't find the container with id 45c3805baef083eb6a926562dbfea57d28d5d2f9dbf4f9b1a4bc58f8a1064f31 Apr 23 18:18:19.637459 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:19.637416 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" event={"ID":"a0c0f2c4-353c-4741-81a8-57f026b023d9","Type":"ContainerStarted","Data":"45c3805baef083eb6a926562dbfea57d28d5d2f9dbf4f9b1a4bc58f8a1064f31"} Apr 23 18:18:24.658510 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:24.658469 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" event={"ID":"a0c0f2c4-353c-4741-81a8-57f026b023d9","Type":"ContainerStarted","Data":"70203067cb30b4d650331267c4e5935a867aca4df67593b305d0ee089da289f6"} Apr 23 18:18:24.675263 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:24.675197 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-mj4k9" podStartSLOduration=1.881036604 podStartE2EDuration="7.675180395s" podCreationTimestamp="2026-04-23 18:18:17 +0000 UTC" firstStartedPulling="2026-04-23 18:18:18.751807151 +0000 UTC m=+2197.771260145" lastFinishedPulling="2026-04-23 18:18:24.545950923 +0000 UTC m=+2203.565403936" observedRunningTime="2026-04-23 18:18:24.674509437 +0000 UTC m=+2203.693962466" watchObservedRunningTime="2026-04-23 18:18:24.675180395 +0000 UTC m=+2203.694633411" Apr 23 18:18:29.640470 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:18:29.640438 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-hhjsv" Apr 23 18:19:01.741457 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.741369 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-52nnx"] Apr 23 18:19:01.769355 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.769316 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-52nnx"] Apr 23 18:19:01.769355 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.769357 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-52nnx"] Apr 23 18:19:01.769561 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.769471 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:01.771697 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.771670 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Apr 23 18:19:01.909888 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.909847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp56c\" (UniqueName: \"kubernetes.io/projected/903d7de4-4f05-4999-bc57-9700e74f64b3-kube-api-access-dp56c\") pod \"limitador-limitador-67566c68b4-52nnx\" (UID: \"903d7de4-4f05-4999-bc57-9700e74f64b3\") " pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:01.910071 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:01.909899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/903d7de4-4f05-4999-bc57-9700e74f64b3-config-file\") pod \"limitador-limitador-67566c68b4-52nnx\" (UID: \"903d7de4-4f05-4999-bc57-9700e74f64b3\") " pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:02.010958 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.010853 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dp56c\" (UniqueName: \"kubernetes.io/projected/903d7de4-4f05-4999-bc57-9700e74f64b3-kube-api-access-dp56c\") pod \"limitador-limitador-67566c68b4-52nnx\" (UID: \"903d7de4-4f05-4999-bc57-9700e74f64b3\") " pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:02.010958 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.010913 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/903d7de4-4f05-4999-bc57-9700e74f64b3-config-file\") pod \"limitador-limitador-67566c68b4-52nnx\" (UID: \"903d7de4-4f05-4999-bc57-9700e74f64b3\") " pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:02.011566 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.011544 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/903d7de4-4f05-4999-bc57-9700e74f64b3-config-file\") pod \"limitador-limitador-67566c68b4-52nnx\" (UID: \"903d7de4-4f05-4999-bc57-9700e74f64b3\") " pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:02.022751 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.022717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp56c\" (UniqueName: \"kubernetes.io/projected/903d7de4-4f05-4999-bc57-9700e74f64b3-kube-api-access-dp56c\") pod \"limitador-limitador-67566c68b4-52nnx\" (UID: \"903d7de4-4f05-4999-bc57-9700e74f64b3\") " pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:02.080348 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.080289 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:02.180919 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.180304 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-55pqx"] Apr 23 18:19:02.185220 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.185195 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:02.188304 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.188279 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vnrmc\"" Apr 23 18:19:02.191990 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.191617 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-55pqx"] Apr 23 18:19:02.219841 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.219811 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-52nnx"] Apr 23 18:19:02.222043 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:19:02.222008 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903d7de4_4f05_4999_bc57_9700e74f64b3.slice/crio-1e89b59385b447ab6452c760d502dfdc9b0097df6db6967f82dd263a0f65611f WatchSource:0}: Error finding container 1e89b59385b447ab6452c760d502dfdc9b0097df6db6967f82dd263a0f65611f: Status 404 returned error can't find the container with id 1e89b59385b447ab6452c760d502dfdc9b0097df6db6967f82dd263a0f65611f Apr 23 18:19:02.313288 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.313191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzdb\" (UniqueName: \"kubernetes.io/projected/aace7723-2f51-4e04-a6fd-7d671a096fb8-kube-api-access-ggzdb\") pod \"authorino-674b59b84c-55pqx\" (UID: \"aace7723-2f51-4e04-a6fd-7d671a096fb8\") " pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:02.414084 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.414045 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzdb\" (UniqueName: \"kubernetes.io/projected/aace7723-2f51-4e04-a6fd-7d671a096fb8-kube-api-access-ggzdb\") pod \"authorino-674b59b84c-55pqx\" (UID: \"aace7723-2f51-4e04-a6fd-7d671a096fb8\") " pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:02.422936 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.422897 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzdb\" (UniqueName: \"kubernetes.io/projected/aace7723-2f51-4e04-a6fd-7d671a096fb8-kube-api-access-ggzdb\") pod \"authorino-674b59b84c-55pqx\" (UID: \"aace7723-2f51-4e04-a6fd-7d671a096fb8\") " pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:02.497834 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.497798 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:02.621295 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.621120 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-55pqx"] Apr 23 18:19:02.623847 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:19:02.623807 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaace7723_2f51_4e04_a6fd_7d671a096fb8.slice/crio-18a11e8e7dae5c6352cb7964fd8694c583d63dfddc7db09b5ccf384673638373 WatchSource:0}: Error finding container 18a11e8e7dae5c6352cb7964fd8694c583d63dfddc7db09b5ccf384673638373: Status 404 returned error can't find the container with id 18a11e8e7dae5c6352cb7964fd8694c583d63dfddc7db09b5ccf384673638373 Apr 23 18:19:02.789189 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.789146 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-55pqx" event={"ID":"aace7723-2f51-4e04-a6fd-7d671a096fb8","Type":"ContainerStarted","Data":"18a11e8e7dae5c6352cb7964fd8694c583d63dfddc7db09b5ccf384673638373"} Apr 23 18:19:02.790256 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:02.790226 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" event={"ID":"903d7de4-4f05-4999-bc57-9700e74f64b3","Type":"ContainerStarted","Data":"1e89b59385b447ab6452c760d502dfdc9b0097df6db6967f82dd263a0f65611f"} Apr 23 18:19:03.798562 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:03.798511 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" event={"ID":"903d7de4-4f05-4999-bc57-9700e74f64b3","Type":"ContainerStarted","Data":"73a868c4927d4843b45078fa119dcc53ad0e3481cb2aeaf30186de6c276a2c74"} Apr 23 18:19:03.799056 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:03.798652 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:05.806715 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:05.806616 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-55pqx" event={"ID":"aace7723-2f51-4e04-a6fd-7d671a096fb8","Type":"ContainerStarted","Data":"7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5"} Apr 23 18:19:05.824376 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:05.824319 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" podStartSLOduration=3.7089944089999998 podStartE2EDuration="4.824301077s" podCreationTimestamp="2026-04-23 18:19:01 +0000 UTC" firstStartedPulling="2026-04-23 18:19:02.224096654 +0000 UTC m=+2241.243549648" lastFinishedPulling="2026-04-23 18:19:03.339403314 +0000 UTC m=+2242.358856316" observedRunningTime="2026-04-23 18:19:03.82280673 +0000 UTC m=+2242.842259747" watchObservedRunningTime="2026-04-23 18:19:05.824301077 +0000 UTC m=+2244.843754095" Apr 23 18:19:05.824643 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:05.824614 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-55pqx" podStartSLOduration=0.950310816 podStartE2EDuration="3.824609004s" podCreationTimestamp="2026-04-23 18:19:02 +0000 UTC" firstStartedPulling="2026-04-23 18:19:02.625130282 +0000 UTC m=+2241.644583277" lastFinishedPulling="2026-04-23 18:19:05.499428466 +0000 UTC m=+2244.518881465" observedRunningTime="2026-04-23 18:19:05.824044147 +0000 UTC m=+2244.843497162" watchObservedRunningTime="2026-04-23 18:19:05.824609004 +0000 UTC m=+2244.844062012" Apr 23 18:19:06.244619 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:06.244561 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-55pqx"] Apr 23 18:19:07.813352 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:07.813299 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-55pqx" podUID="aace7723-2f51-4e04-a6fd-7d671a096fb8" containerName="authorino" containerID="cri-o://7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5" gracePeriod=30 Apr 23 18:19:08.063625 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.063529 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:08.070256 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.070225 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggzdb\" (UniqueName: \"kubernetes.io/projected/aace7723-2f51-4e04-a6fd-7d671a096fb8-kube-api-access-ggzdb\") pod \"aace7723-2f51-4e04-a6fd-7d671a096fb8\" (UID: \"aace7723-2f51-4e04-a6fd-7d671a096fb8\") " Apr 23 18:19:08.072458 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.072428 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aace7723-2f51-4e04-a6fd-7d671a096fb8-kube-api-access-ggzdb" (OuterVolumeSpecName: "kube-api-access-ggzdb") pod "aace7723-2f51-4e04-a6fd-7d671a096fb8" (UID: "aace7723-2f51-4e04-a6fd-7d671a096fb8"). InnerVolumeSpecName "kube-api-access-ggzdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 18:19:08.171184 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.171141 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzdb\" (UniqueName: \"kubernetes.io/projected/aace7723-2f51-4e04-a6fd-7d671a096fb8-kube-api-access-ggzdb\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:19:08.818283 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.818242 2575 generic.go:358] "Generic (PLEG): container finished" podID="aace7723-2f51-4e04-a6fd-7d671a096fb8" containerID="7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5" exitCode=0 Apr 23 18:19:08.818705 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.818297 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-55pqx" Apr 23 18:19:08.818705 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.818335 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-55pqx" event={"ID":"aace7723-2f51-4e04-a6fd-7d671a096fb8","Type":"ContainerDied","Data":"7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5"} Apr 23 18:19:08.818705 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.818378 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-55pqx" event={"ID":"aace7723-2f51-4e04-a6fd-7d671a096fb8","Type":"ContainerDied","Data":"18a11e8e7dae5c6352cb7964fd8694c583d63dfddc7db09b5ccf384673638373"} Apr 23 18:19:08.818705 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.818394 2575 scope.go:117] "RemoveContainer" containerID="7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5" Apr 23 18:19:08.827408 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.827384 2575 scope.go:117] "RemoveContainer" containerID="7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5" Apr 23 18:19:08.827682 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:19:08.827660 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5\": container with ID starting with 7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5 not found: ID does not exist" containerID="7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5" Apr 23 18:19:08.827787 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.827689 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5"} err="failed to get container status \"7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5\": rpc error: code = NotFound desc = could not find container \"7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5\": container with ID starting with 7173d80e847868cf0528444b6abb51a9c1a162b3a36f41b49bbb316b333b32b5 not found: ID does not exist" Apr 23 18:19:08.839430 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.839400 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-55pqx"] Apr 23 18:19:08.841640 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:08.841609 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-55pqx"] Apr 23 18:19:09.594619 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:09.594567 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aace7723-2f51-4e04-a6fd-7d671a096fb8" path="/var/lib/kubelet/pods/aace7723-2f51-4e04-a6fd-7d671a096fb8/volumes" Apr 23 18:19:14.803040 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:14.803008 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-52nnx" Apr 23 18:19:24.354676 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.354632 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-vlxjh"] Apr 23 18:19:24.355069 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.354968 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aace7723-2f51-4e04-a6fd-7d671a096fb8" containerName="authorino" Apr 23 18:19:24.355069 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.354981 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aace7723-2f51-4e04-a6fd-7d671a096fb8" containerName="authorino" Apr 23 18:19:24.355069 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.355049 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="aace7723-2f51-4e04-a6fd-7d671a096fb8" containerName="authorino" Apr 23 18:19:24.359270 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.359251 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.361772 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.361750 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Apr 23 18:19:24.362690 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.362672 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vnrmc\"" Apr 23 18:19:24.370076 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.370050 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-vlxjh"] Apr 23 18:19:24.410282 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.410242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4lv8\" (UniqueName: \"kubernetes.io/projected/a7814e01-e9c6-44f6-996c-a7514e3ab922-kube-api-access-m4lv8\") pod \"authorino-68bd676465-vlxjh\" (UID: \"a7814e01-e9c6-44f6-996c-a7514e3ab922\") " pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.410467 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.410315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7814e01-e9c6-44f6-996c-a7514e3ab922-tls-cert\") pod \"authorino-68bd676465-vlxjh\" (UID: \"a7814e01-e9c6-44f6-996c-a7514e3ab922\") " pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.511736 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.511695 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lv8\" (UniqueName: \"kubernetes.io/projected/a7814e01-e9c6-44f6-996c-a7514e3ab922-kube-api-access-m4lv8\") pod \"authorino-68bd676465-vlxjh\" (UID: \"a7814e01-e9c6-44f6-996c-a7514e3ab922\") " pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.511915 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.511776 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7814e01-e9c6-44f6-996c-a7514e3ab922-tls-cert\") pod \"authorino-68bd676465-vlxjh\" (UID: \"a7814e01-e9c6-44f6-996c-a7514e3ab922\") " pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.514186 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.514155 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7814e01-e9c6-44f6-996c-a7514e3ab922-tls-cert\") pod \"authorino-68bd676465-vlxjh\" (UID: \"a7814e01-e9c6-44f6-996c-a7514e3ab922\") " pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.528129 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.528104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lv8\" (UniqueName: \"kubernetes.io/projected/a7814e01-e9c6-44f6-996c-a7514e3ab922-kube-api-access-m4lv8\") pod \"authorino-68bd676465-vlxjh\" (UID: \"a7814e01-e9c6-44f6-996c-a7514e3ab922\") " pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.669112 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.669014 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-vlxjh" Apr 23 18:19:24.799488 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.799461 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-vlxjh"] Apr 23 18:19:24.802045 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:19:24.802013 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7814e01_e9c6_44f6_996c_a7514e3ab922.slice/crio-f24ddf512603ca15d8a20cba7b26f71ac9a3079fe90d752ebbaab0895018f941 WatchSource:0}: Error finding container f24ddf512603ca15d8a20cba7b26f71ac9a3079fe90d752ebbaab0895018f941: Status 404 returned error can't find the container with id f24ddf512603ca15d8a20cba7b26f71ac9a3079fe90d752ebbaab0895018f941 Apr 23 18:19:24.873332 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:24.873285 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-vlxjh" event={"ID":"a7814e01-e9c6-44f6-996c-a7514e3ab922","Type":"ContainerStarted","Data":"f24ddf512603ca15d8a20cba7b26f71ac9a3079fe90d752ebbaab0895018f941"} Apr 23 18:19:25.878227 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:25.878120 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-vlxjh" event={"ID":"a7814e01-e9c6-44f6-996c-a7514e3ab922","Type":"ContainerStarted","Data":"15ddb50fcce8c14ecd1b92a1d9fb34054740e125a6d1dc0483673a287f1d4d52"} Apr 23 18:19:25.897757 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:19:25.897704 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-vlxjh" podStartSLOduration=1.138685564 podStartE2EDuration="1.897688794s" podCreationTimestamp="2026-04-23 18:19:24 +0000 UTC" firstStartedPulling="2026-04-23 18:19:24.806133028 +0000 UTC m=+2263.825586037" lastFinishedPulling="2026-04-23 18:19:25.565136271 +0000 UTC m=+2264.584589267" observedRunningTime="2026-04-23 18:19:25.895977476 +0000 UTC m=+2264.915430539" watchObservedRunningTime="2026-04-23 18:19:25.897688794 +0000 UTC m=+2264.917141807" Apr 23 18:21:41.579793 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:41.579673 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:21:41.583330 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:41.581831 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:21:52.732113 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.732073 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r"] Apr 23 18:21:52.735289 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.735270 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.737531 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.737496 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Apr 23 18:21:52.737531 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.737523 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-78dl4\"" Apr 23 18:21:52.737735 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.737610 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 23 18:21:52.737735 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.737611 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Apr 23 18:21:52.744088 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.744061 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r"] Apr 23 18:21:52.876496 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.876496 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.876768 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.876768 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.876768 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.876768 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.876768 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.876661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcld6\" (UniqueName: \"kubernetes.io/projected/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kube-api-access-dcld6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.977821 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.977784 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.977821 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.977830 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.977964 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978024 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978058 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978253 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978093 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978253 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978124 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dcld6\" (UniqueName: \"kubernetes.io/projected/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kube-api-access-dcld6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978356 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978335 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978409 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978467 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978446 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.978551 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.978533 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.980300 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.980278 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.980632 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.980613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:52.986211 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:52.986143 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcld6\" (UniqueName: \"kubernetes.io/projected/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kube-api-access-dcld6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:53.046953 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:53.046911 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:21:53.182521 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:53.182477 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r"] Apr 23 18:21:53.185306 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:21:53.185271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaea6034_670d_4afd_9c3c_bbf1ae8d0780.slice/crio-f1e31fdd8635fb49b98d4b17cd59026b2543db53800aceb9d6efbb186da81f3c WatchSource:0}: Error finding container f1e31fdd8635fb49b98d4b17cd59026b2543db53800aceb9d6efbb186da81f3c: Status 404 returned error can't find the container with id f1e31fdd8635fb49b98d4b17cd59026b2543db53800aceb9d6efbb186da81f3c Apr 23 18:21:53.187350 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:53.187331 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 18:21:53.380548 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:53.380455 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" event={"ID":"faea6034-670d-4afd-9c3c-bbf1ae8d0780","Type":"ContainerStarted","Data":"f1e31fdd8635fb49b98d4b17cd59026b2543db53800aceb9d6efbb186da81f3c"} Apr 23 18:21:57.399696 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:21:57.399654 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" event={"ID":"faea6034-670d-4afd-9c3c-bbf1ae8d0780","Type":"ContainerStarted","Data":"49fbbd476a138282aba9c4bfd05e1d61b24310f526e7089f47e9b9ca56261a11"} Apr 23 18:22:00.411724 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:00.411687 2575 generic.go:358] "Generic (PLEG): container finished" podID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerID="49fbbd476a138282aba9c4bfd05e1d61b24310f526e7089f47e9b9ca56261a11" exitCode=0 Apr 23 18:22:00.412067 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:00.411743 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" event={"ID":"faea6034-670d-4afd-9c3c-bbf1ae8d0780","Type":"ContainerDied","Data":"49fbbd476a138282aba9c4bfd05e1d61b24310f526e7089f47e9b9ca56261a11"} Apr 23 18:22:02.420989 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:02.420944 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" event={"ID":"faea6034-670d-4afd-9c3c-bbf1ae8d0780","Type":"ContainerStarted","Data":"4368b925ce76c585fd855e921e8db086d34efe94aa07f4860cf31df644b8eb30"} Apr 23 18:22:02.443657 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:02.443604 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" podStartSLOduration=2.076014566 podStartE2EDuration="10.443573546s" podCreationTimestamp="2026-04-23 18:21:52 +0000 UTC" firstStartedPulling="2026-04-23 18:21:53.187456293 +0000 UTC m=+2412.206909288" lastFinishedPulling="2026-04-23 18:22:01.55501527 +0000 UTC m=+2420.574468268" observedRunningTime="2026-04-23 18:22:02.442335922 +0000 UTC m=+2421.461788938" watchObservedRunningTime="2026-04-23 18:22:02.443573546 +0000 UTC m=+2421.463026561" Apr 23 18:22:03.047086 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:03.047042 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:22:03.047086 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:03.047082 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:22:03.060065 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:03.060029 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:22:03.436151 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:03.436067 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:22:17.078905 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:17.078865 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r"] Apr 23 18:22:17.079423 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:17.079161 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerName="main" containerID="cri-o://4368b925ce76c585fd855e921e8db086d34efe94aa07f4860cf31df644b8eb30" gracePeriod=30 Apr 23 18:22:17.473543 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:17.473446 2575 generic.go:358] "Generic (PLEG): container finished" podID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerID="4368b925ce76c585fd855e921e8db086d34efe94aa07f4860cf31df644b8eb30" exitCode=0 Apr 23 18:22:17.473543 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:17.473520 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" event={"ID":"faea6034-670d-4afd-9c3c-bbf1ae8d0780","Type":"ContainerDied","Data":"4368b925ce76c585fd855e921e8db086d34efe94aa07f4860cf31df644b8eb30"} Apr 23 18:22:17.864287 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:17.864263 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:22:18.003177 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003140 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcld6\" (UniqueName: \"kubernetes.io/projected/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kube-api-access-dcld6\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003355 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003192 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tls-certs\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003355 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003234 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-model-cache\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003355 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003272 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tmp-dir\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003517 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003492 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kserve-provision-location\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003571 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003527 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:18.003571 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003559 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-dshm\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003703 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003632 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-home\") pod \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\" (UID: \"faea6034-670d-4afd-9c3c-bbf1ae8d0780\") " Apr 23 18:22:18.003703 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003534 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-model-cache" (OuterVolumeSpecName: "model-cache") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:18.003938 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003914 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:18.003938 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003936 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-home" (OuterVolumeSpecName: "home") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:18.004135 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.003942 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:18.005665 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.005637 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-dshm" (OuterVolumeSpecName: "dshm") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:18.005749 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.005710 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kube-api-access-dcld6" (OuterVolumeSpecName: "kube-api-access-dcld6") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "kube-api-access-dcld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 18:22:18.005830 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.005810 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 18:22:18.104442 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.104357 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcld6\" (UniqueName: \"kubernetes.io/projected/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kube-api-access-dcld6\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:18.104442 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.104389 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faea6034-670d-4afd-9c3c-bbf1ae8d0780-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:18.104442 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.104399 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:18.104442 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.104408 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:18.478749 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.478718 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" Apr 23 18:22:18.478945 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.478716 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r" event={"ID":"faea6034-670d-4afd-9c3c-bbf1ae8d0780","Type":"ContainerDied","Data":"f1e31fdd8635fb49b98d4b17cd59026b2543db53800aceb9d6efbb186da81f3c"} Apr 23 18:22:18.478945 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.478842 2575 scope.go:117] "RemoveContainer" containerID="4368b925ce76c585fd855e921e8db086d34efe94aa07f4860cf31df644b8eb30" Apr 23 18:22:18.487154 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:18.487137 2575 scope.go:117] "RemoveContainer" containerID="49fbbd476a138282aba9c4bfd05e1d61b24310f526e7089f47e9b9ca56261a11" Apr 23 18:22:19.062470 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:19.062408 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "faea6034-670d-4afd-9c3c-bbf1ae8d0780" (UID: "faea6034-670d-4afd-9c3c-bbf1ae8d0780"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:19.101707 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:19.101667 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r"] Apr 23 18:22:19.106610 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:19.106568 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-5db5b6c7cccrw6r"] Apr 23 18:22:19.114165 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:19.114139 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faea6034-670d-4afd-9c3c-bbf1ae8d0780-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:19.595572 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:19.595538 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" path="/var/lib/kubelet/pods/faea6034-670d-4afd-9c3c-bbf1ae8d0780/volumes" Apr 23 18:22:29.250404 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.250367 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn"] Apr 23 18:22:29.250905 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.250734 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerName="main" Apr 23 18:22:29.250905 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.250745 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerName="main" Apr 23 18:22:29.250905 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.250761 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerName="storage-initializer" Apr 23 18:22:29.250905 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.250766 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerName="storage-initializer" Apr 23 18:22:29.250905 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.250830 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="faea6034-670d-4afd-9c3c-bbf1ae8d0780" containerName="main" Apr 23 18:22:29.255647 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.255621 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.258421 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.258395 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Apr 23 18:22:29.258566 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.258444 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-78dl4\"" Apr 23 18:22:29.258566 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.258444 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Apr 23 18:22:29.258566 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.258446 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 23 18:22:29.263110 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.262904 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn"] Apr 23 18:22:29.408939 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.408900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b89q\" (UniqueName: \"kubernetes.io/projected/2632c010-8311-4037-81d8-71026a98eaf2-kube-api-access-8b89q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.409116 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.408949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.409116 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.408972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.409116 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.409018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.409116 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.409042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.409116 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.409087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.409280 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.409159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2632c010-8311-4037-81d8-71026a98eaf2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510303 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510210 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2632c010-8311-4037-81d8-71026a98eaf2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510303 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510267 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8b89q\" (UniqueName: \"kubernetes.io/projected/2632c010-8311-4037-81d8-71026a98eaf2-kube-api-access-8b89q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510303 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510297 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510581 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510314 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510581 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510344 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510581 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510370 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510581 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510390 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510824 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510800 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510883 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510883 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.510990 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.510957 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.512682 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.512653 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.512883 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.512868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2632c010-8311-4037-81d8-71026a98eaf2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.519183 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.519156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b89q\" (UniqueName: \"kubernetes.io/projected/2632c010-8311-4037-81d8-71026a98eaf2-kube-api-access-8b89q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.567205 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.567165 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:29.713526 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:29.713406 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn"] Apr 23 18:22:29.716451 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:22:29.716422 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2632c010_8311_4037_81d8_71026a98eaf2.slice/crio-7fa241f5699865a0c8b967ce72eb627026586e89afbd3282bfa0c3ade7cdae8a WatchSource:0}: Error finding container 7fa241f5699865a0c8b967ce72eb627026586e89afbd3282bfa0c3ade7cdae8a: Status 404 returned error can't find the container with id 7fa241f5699865a0c8b967ce72eb627026586e89afbd3282bfa0c3ade7cdae8a Apr 23 18:22:30.522302 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:30.522257 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" event={"ID":"2632c010-8311-4037-81d8-71026a98eaf2","Type":"ContainerStarted","Data":"2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566"} Apr 23 18:22:30.522302 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:30.522305 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" event={"ID":"2632c010-8311-4037-81d8-71026a98eaf2","Type":"ContainerStarted","Data":"7fa241f5699865a0c8b967ce72eb627026586e89afbd3282bfa0c3ade7cdae8a"} Apr 23 18:22:32.524525 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:32.524489 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn"] Apr 23 18:22:32.528197 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:32.528162 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" podUID="2632c010-8311-4037-81d8-71026a98eaf2" containerName="storage-initializer" containerID="cri-o://2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566" gracePeriod=30 Apr 23 18:22:33.987500 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:33.987474 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:34.154445 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154351 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b89q\" (UniqueName: \"kubernetes.io/projected/2632c010-8311-4037-81d8-71026a98eaf2-kube-api-access-8b89q\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154445 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154408 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-model-cache\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154445 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154434 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-dshm\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154763 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154479 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2632c010-8311-4037-81d8-71026a98eaf2-tls-certs\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154763 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154506 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-home\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154763 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154547 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-tmp-dir\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154763 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154622 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-kserve-provision-location\") pod \"2632c010-8311-4037-81d8-71026a98eaf2\" (UID: \"2632c010-8311-4037-81d8-71026a98eaf2\") " Apr 23 18:22:34.154976 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154781 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:34.154976 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154820 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-home" (OuterVolumeSpecName: "home") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:34.154976 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154915 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.154976 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154927 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.154976 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.154941 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:34.156708 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.156685 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2632c010-8311-4037-81d8-71026a98eaf2-kube-api-access-8b89q" (OuterVolumeSpecName: "kube-api-access-8b89q") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "kube-api-access-8b89q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 18:22:34.157134 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.157106 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-dshm" (OuterVolumeSpecName: "dshm") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:34.157246 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.157113 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2632c010-8311-4037-81d8-71026a98eaf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 18:22:34.221569 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.221511 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2632c010-8311-4037-81d8-71026a98eaf2" (UID: "2632c010-8311-4037-81d8-71026a98eaf2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:22:34.256042 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.256012 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.256042 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.256044 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2632c010-8311-4037-81d8-71026a98eaf2-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.256042 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.256056 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.256204 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.256066 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2632c010-8311-4037-81d8-71026a98eaf2-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.256204 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.256077 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b89q\" (UniqueName: \"kubernetes.io/projected/2632c010-8311-4037-81d8-71026a98eaf2-kube-api-access-8b89q\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:22:34.538507 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.538470 2575 generic.go:358] "Generic (PLEG): container finished" podID="2632c010-8311-4037-81d8-71026a98eaf2" containerID="2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566" exitCode=0 Apr 23 18:22:34.538735 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.538539 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" Apr 23 18:22:34.538735 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.538562 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" event={"ID":"2632c010-8311-4037-81d8-71026a98eaf2","Type":"ContainerDied","Data":"2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566"} Apr 23 18:22:34.538735 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.538641 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn" event={"ID":"2632c010-8311-4037-81d8-71026a98eaf2","Type":"ContainerDied","Data":"7fa241f5699865a0c8b967ce72eb627026586e89afbd3282bfa0c3ade7cdae8a"} Apr 23 18:22:34.538735 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.538668 2575 scope.go:117] "RemoveContainer" containerID="2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566" Apr 23 18:22:34.585464 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.585418 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn"] Apr 23 18:22:34.588681 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.588638 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-59f6ff894bf5hzn"] Apr 23 18:22:34.612524 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.612499 2575 scope.go:117] "RemoveContainer" containerID="2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566" Apr 23 18:22:34.612885 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:22:34.612863 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566\": container with ID starting with 2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566 not found: ID does not exist" containerID="2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566" Apr 23 18:22:34.612962 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:34.612897 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566"} err="failed to get container status \"2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566\": rpc error: code = NotFound desc = could not find container \"2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566\": container with ID starting with 2309d7a9d38af14af5aff0cb42f44da7bae24ce5529b352f8a2a8ebac2d97566 not found: ID does not exist" Apr 23 18:22:35.594828 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:35.594795 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2632c010-8311-4037-81d8-71026a98eaf2" path="/var/lib/kubelet/pods/2632c010-8311-4037-81d8-71026a98eaf2/volumes" Apr 23 18:22:38.834578 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.834539 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk"] Apr 23 18:22:38.835043 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.834941 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2632c010-8311-4037-81d8-71026a98eaf2" containerName="storage-initializer" Apr 23 18:22:38.835043 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.834958 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2632c010-8311-4037-81d8-71026a98eaf2" containerName="storage-initializer" Apr 23 18:22:38.835043 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.835013 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="2632c010-8311-4037-81d8-71026a98eaf2" containerName="storage-initializer" Apr 23 18:22:38.839509 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.839488 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.842608 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.842567 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 23 18:22:38.842751 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.842633 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Apr 23 18:22:38.842751 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.842655 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-78dl4\"" Apr 23 18:22:38.842963 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.842942 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\"" Apr 23 18:22:38.850034 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.850010 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk"] Apr 23 18:22:38.892246 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358405dd-d70d-43a9-b1ac-df6be7a6557f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.892246 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gsgg\" (UniqueName: \"kubernetes.io/projected/358405dd-d70d-43a9-b1ac-df6be7a6557f-kube-api-access-9gsgg\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.892473 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.892473 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.892473 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.892651 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.892651 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.892501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993144 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993111 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993144 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993151 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993402 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993181 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358405dd-d70d-43a9-b1ac-df6be7a6557f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993402 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993372 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gsgg\" (UniqueName: \"kubernetes.io/projected/358405dd-d70d-43a9-b1ac-df6be7a6557f-kube-api-access-9gsgg\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993518 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993444 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993613 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993785 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.993967 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.993946 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.994080 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.994053 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.994216 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.994190 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.994470 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.994338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.995997 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.995976 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358405dd-d70d-43a9-b1ac-df6be7a6557f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:38.996222 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:38.996205 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:39.000662 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:39.000642 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gsgg\" (UniqueName: \"kubernetes.io/projected/358405dd-d70d-43a9-b1ac-df6be7a6557f-kube-api-access-9gsgg\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:39.151265 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:39.151164 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:39.278459 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:39.278431 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk"] Apr 23 18:22:39.281294 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:22:39.281264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358405dd_d70d_43a9_b1ac_df6be7a6557f.slice/crio-da8186f31ac0f0c8d5d5b2bfe888b6b86a1ff0b0c8ebca76f4493b4b6b5391b9 WatchSource:0}: Error finding container da8186f31ac0f0c8d5d5b2bfe888b6b86a1ff0b0c8ebca76f4493b4b6b5391b9: Status 404 returned error can't find the container with id da8186f31ac0f0c8d5d5b2bfe888b6b86a1ff0b0c8ebca76f4493b4b6b5391b9 Apr 23 18:22:39.559423 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:39.559383 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" event={"ID":"358405dd-d70d-43a9-b1ac-df6be7a6557f","Type":"ContainerStarted","Data":"f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d"} Apr 23 18:22:39.559423 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:39.559424 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" event={"ID":"358405dd-d70d-43a9-b1ac-df6be7a6557f","Type":"ContainerStarted","Data":"da8186f31ac0f0c8d5d5b2bfe888b6b86a1ff0b0c8ebca76f4493b4b6b5391b9"} Apr 23 18:22:43.575973 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:43.575929 2575 generic.go:358] "Generic (PLEG): container finished" podID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerID="f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d" exitCode=0 Apr 23 18:22:43.576363 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:43.576002 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" event={"ID":"358405dd-d70d-43a9-b1ac-df6be7a6557f","Type":"ContainerDied","Data":"f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d"} Apr 23 18:22:44.581265 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:44.581230 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" event={"ID":"358405dd-d70d-43a9-b1ac-df6be7a6557f","Type":"ContainerStarted","Data":"1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a"} Apr 23 18:22:44.602787 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:44.602742 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" podStartSLOduration=6.602726501 podStartE2EDuration="6.602726501s" podCreationTimestamp="2026-04-23 18:22:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 18:22:44.600810787 +0000 UTC m=+2463.620263799" watchObservedRunningTime="2026-04-23 18:22:44.602726501 +0000 UTC m=+2463.622179516" Apr 23 18:22:49.152350 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:49.152287 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:49.152350 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:49.152365 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:49.165063 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:49.165033 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:22:49.609169 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:22:49.609135 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:26:41.603387 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:26:41.603275 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:26:41.607220 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:26:41.605140 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:31:41.625920 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:31:41.625805 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:31:41.630130 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:31:41.628355 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:36:41.649254 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:36:41.649136 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:36:41.653078 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:36:41.652349 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:37:34.222228 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.222191 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk"] Apr 23 18:37:34.222775 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.222568 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerName="main" containerID="cri-o://1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a" gracePeriod=30 Apr 23 18:37:34.473838 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.473762 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:37:34.606953 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.606916 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-tmp-dir\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607155 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607005 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-dshm\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607155 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607030 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-home\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607155 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607081 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gsgg\" (UniqueName: \"kubernetes.io/projected/358405dd-d70d-43a9-b1ac-df6be7a6557f-kube-api-access-9gsgg\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607155 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607109 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-model-cache\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607155 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607134 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-kserve-provision-location\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607427 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607167 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358405dd-d70d-43a9-b1ac-df6be7a6557f-tls-certs\") pod \"358405dd-d70d-43a9-b1ac-df6be7a6557f\" (UID: \"358405dd-d70d-43a9-b1ac-df6be7a6557f\") " Apr 23 18:37:34.607427 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607262 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:37:34.607427 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607336 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-home" (OuterVolumeSpecName: "home") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:37:34.607550 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607428 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-model-cache" (OuterVolumeSpecName: "model-cache") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:37:34.607550 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607448 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.607550 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.607468 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.609368 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.609343 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358405dd-d70d-43a9-b1ac-df6be7a6557f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 18:37:34.609368 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.609347 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358405dd-d70d-43a9-b1ac-df6be7a6557f-kube-api-access-9gsgg" (OuterVolumeSpecName: "kube-api-access-9gsgg") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "kube-api-access-9gsgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 18:37:34.609545 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.609501 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-dshm" (OuterVolumeSpecName: "dshm") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:37:34.646923 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.646886 2575 generic.go:358] "Generic (PLEG): container finished" podID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerID="1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a" exitCode=0 Apr 23 18:37:34.647087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.646929 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" event={"ID":"358405dd-d70d-43a9-b1ac-df6be7a6557f","Type":"ContainerDied","Data":"1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a"} Apr 23 18:37:34.647087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.646968 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" event={"ID":"358405dd-d70d-43a9-b1ac-df6be7a6557f","Type":"ContainerDied","Data":"da8186f31ac0f0c8d5d5b2bfe888b6b86a1ff0b0c8ebca76f4493b4b6b5391b9"} Apr 23 18:37:34.647087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.646986 2575 scope.go:117] "RemoveContainer" containerID="1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a" Apr 23 18:37:34.647087 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.646982 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk" Apr 23 18:37:34.655904 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.655876 2575 scope.go:117] "RemoveContainer" containerID="f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d" Apr 23 18:37:34.664291 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.664230 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "358405dd-d70d-43a9-b1ac-df6be7a6557f" (UID: "358405dd-d70d-43a9-b1ac-df6be7a6557f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:37:34.708330 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.708284 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.708330 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.708323 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gsgg\" (UniqueName: \"kubernetes.io/projected/358405dd-d70d-43a9-b1ac-df6be7a6557f-kube-api-access-9gsgg\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.708330 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.708336 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.708570 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.708346 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358405dd-d70d-43a9-b1ac-df6be7a6557f-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.708570 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.708356 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358405dd-d70d-43a9-b1ac-df6be7a6557f-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:37:34.717680 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.717656 2575 scope.go:117] "RemoveContainer" containerID="1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a" Apr 23 18:37:34.718010 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:37:34.717989 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a\": container with ID starting with 1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a not found: ID does not exist" containerID="1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a" Apr 23 18:37:34.718086 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.718023 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a"} err="failed to get container status \"1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a\": rpc error: code = NotFound desc = could not find container \"1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a\": container with ID starting with 1a358099b4dc3225ba243c54aefe9133786b9541ac2dee124751416b6144b80a not found: ID does not exist" Apr 23 18:37:34.718086 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.718052 2575 scope.go:117] "RemoveContainer" containerID="f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d" Apr 23 18:37:34.718338 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:37:34.718318 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d\": container with ID starting with f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d not found: ID does not exist" containerID="f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d" Apr 23 18:37:34.718395 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.718344 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d"} err="failed to get container status \"f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d\": rpc error: code = NotFound desc = could not find container \"f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d\": container with ID starting with f38bba4ed2c599b44eea240136ef36554ec2381fc01d43f9714d057a704e3c1d not found: ID does not exist" Apr 23 18:37:34.969610 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.969551 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk"] Apr 23 18:37:34.974172 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:34.974142 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-69b8c75f54sbwdk"] Apr 23 18:37:35.595258 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:37:35.595220 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" path="/var/lib/kubelet/pods/358405dd-d70d-43a9-b1ac-df6be7a6557f/volumes" Apr 23 18:38:02.365513 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.365466 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh"] Apr 23 18:38:02.365973 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.365906 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerName="main" Apr 23 18:38:02.365973 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.365922 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerName="main" Apr 23 18:38:02.365973 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.365932 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerName="storage-initializer" Apr 23 18:38:02.365973 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.365939 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerName="storage-initializer" Apr 23 18:38:02.366103 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.366010 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="358405dd-d70d-43a9-b1ac-df6be7a6557f" containerName="main" Apr 23 18:38:02.369078 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.369060 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.371563 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.371538 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Apr 23 18:38:02.371718 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.371582 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\"" Apr 23 18:38:02.371718 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.371614 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-78dl4\"" Apr 23 18:38:02.372419 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.372400 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 23 18:38:02.379157 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.379126 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh"] Apr 23 18:38:02.448126 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.448329 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.448329 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.448329 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.448462 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.448462 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.448462 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.448410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgnj\" (UniqueName: \"kubernetes.io/projected/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kube-api-access-6bgnj\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.549605 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549554 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.549774 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549638 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.549774 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549669 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.549774 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549687 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.549912 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549811 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgnj\" (UniqueName: \"kubernetes.io/projected/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kube-api-access-6bgnj\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.549912 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549881 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.550014 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.549913 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.550067 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.550024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.550128 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.550071 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.550277 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.550246 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.550425 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.550383 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.552153 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.552114 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.552444 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.552422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.558489 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.558460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgnj\" (UniqueName: \"kubernetes.io/projected/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kube-api-access-6bgnj\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.680022 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.679919 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:38:02.817167 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.817125 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh"] Apr 23 18:38:02.820899 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:38:02.820868 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3b04bc_90d0_46a2_a4b5_a17dadd7652e.slice/crio-4380fd68976f8111edf342c9a49d02501bcf60b45061d1acd47b12b5d471f620 WatchSource:0}: Error finding container 4380fd68976f8111edf342c9a49d02501bcf60b45061d1acd47b12b5d471f620: Status 404 returned error can't find the container with id 4380fd68976f8111edf342c9a49d02501bcf60b45061d1acd47b12b5d471f620 Apr 23 18:38:02.822681 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:02.822662 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 18:38:03.749700 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:03.749655 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" event={"ID":"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e","Type":"ContainerStarted","Data":"31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444"} Apr 23 18:38:03.749700 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:03.749704 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" event={"ID":"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e","Type":"ContainerStarted","Data":"4380fd68976f8111edf342c9a49d02501bcf60b45061d1acd47b12b5d471f620"} Apr 23 18:38:07.764312 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:07.764275 2575 generic.go:358] "Generic (PLEG): container finished" podID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerID="31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444" exitCode=0 Apr 23 18:38:07.764771 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:07.764353 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" event={"ID":"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e","Type":"ContainerDied","Data":"31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444"} Apr 23 18:38:53.938321 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:53.938279 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" event={"ID":"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e","Type":"ContainerStarted","Data":"7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d"} Apr 23 18:38:53.962339 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:38:53.962283 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podStartSLOduration=6.797865228 podStartE2EDuration="51.962268712s" podCreationTimestamp="2026-04-23 18:38:02 +0000 UTC" firstStartedPulling="2026-04-23 18:38:07.765675831 +0000 UTC m=+3386.785128825" lastFinishedPulling="2026-04-23 18:38:52.930079316 +0000 UTC m=+3431.949532309" observedRunningTime="2026-04-23 18:38:53.960966074 +0000 UTC m=+3432.980419100" watchObservedRunningTime="2026-04-23 18:38:53.962268712 +0000 UTC m=+3432.981721727" Apr 23 18:39:02.681072 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:02.681027 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:39:02.681700 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:02.681096 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:39:02.682624 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:02.682566 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:39:12.680955 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:12.680892 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:39:22.680661 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:22.680608 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:39:32.681244 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:32.681181 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:39:42.680470 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:42.680426 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:39:52.680781 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:39:52.680733 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:40:02.680692 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:40:02.680579 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:40:12.680993 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:40:12.680944 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:40:22.681333 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:40:22.681280 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused" Apr 23 18:40:32.690512 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:40:32.690467 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:40:32.698548 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:40:32.698517 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:41:41.672622 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:41:41.672491 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:41:41.677536 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:41:41.677514 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:46:41.695094 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:46:41.694982 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:46:41.701828 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:46:41.701808 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:51:41.718482 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:51:41.718360 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:51:41.724473 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:51:41.724454 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:56:41.740742 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:56:41.740714 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:56:41.748883 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:56:41.748861 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 18:58:57.641951 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:58:57.641917 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh"] Apr 23 18:58:57.642461 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:58:57.642276 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" containerID="cri-o://7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d" gracePeriod=30 Apr 23 18:59:13.752985 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.752956 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2"] Apr 23 18:59:13.756275 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.756255 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.758359 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.758339 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" Apr 23 18:59:13.765259 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.765224 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2"] Apr 23 18:59:13.809368 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1bae63-62aa-43d7-b009-a2b70953fd84-tls-certs\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.809535 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-dshm\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.809535 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-model-cache\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.809535 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-home\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.809535 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-tmp-dir\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.809693 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.809693 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.809632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7gx\" (UniqueName: \"kubernetes.io/projected/7b1bae63-62aa-43d7-b009-a2b70953fd84-kube-api-access-kr7gx\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.910814 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.910774 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-dshm\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.910814 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.910817 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-model-cache\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911046 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.910843 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-home\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911046 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.910886 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-tmp-dir\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911046 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.910922 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911046 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.910952 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7gx\" (UniqueName: \"kubernetes.io/projected/7b1bae63-62aa-43d7-b009-a2b70953fd84-kube-api-access-kr7gx\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911046 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.911030 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1bae63-62aa-43d7-b009-a2b70953fd84-tls-certs\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911321 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.911297 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-model-cache\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911397 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.911341 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-home\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911397 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.911375 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-tmp-dir\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.911499 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.911431 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.913242 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.913217 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-dshm\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.913368 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.913354 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1bae63-62aa-43d7-b009-a2b70953fd84-tls-certs\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:13.918748 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:13.918731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7gx\" (UniqueName: \"kubernetes.io/projected/7b1bae63-62aa-43d7-b009-a2b70953fd84-kube-api-access-kr7gx\") pod \"custom-route-timeout-test-kserve-7cc767b4f4-wpnr2\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:14.067884 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:14.067801 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 18:59:14.194724 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:14.194664 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2"] Apr 23 18:59:14.197684 ip-10-0-137-68 kubenswrapper[2575]: W0423 18:59:14.197646 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1bae63_62aa_43d7_b009_a2b70953fd84.slice/crio-be46efa6f20e32e2810bada890981443dd67c35db408aabbb75033c01a3c9033 WatchSource:0}: Error finding container be46efa6f20e32e2810bada890981443dd67c35db408aabbb75033c01a3c9033: Status 404 returned error can't find the container with id be46efa6f20e32e2810bada890981443dd67c35db408aabbb75033c01a3c9033 Apr 23 18:59:14.199893 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:14.199876 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 18:59:15.095975 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:15.095936 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" event={"ID":"7b1bae63-62aa-43d7-b009-a2b70953fd84","Type":"ContainerStarted","Data":"fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef"} Apr 23 18:59:15.096371 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:15.095982 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" event={"ID":"7b1bae63-62aa-43d7-b009-a2b70953fd84","Type":"ContainerStarted","Data":"be46efa6f20e32e2810bada890981443dd67c35db408aabbb75033c01a3c9033"} Apr 23 18:59:27.890896 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.890869 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh_5b3b04bc-90d0-46a2-a4b5-a17dadd7652e/main/0.log" Apr 23 18:59:27.891277 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.891234 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:59:27.930156 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930080 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tls-certs\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930156 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930125 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kserve-provision-location\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930156 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930145 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tmp-dir\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930418 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930273 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bgnj\" (UniqueName: \"kubernetes.io/projected/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kube-api-access-6bgnj\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930418 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930343 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-model-cache\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930418 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930385 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-dshm\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930565 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930443 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-home\") pod \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\" (UID: \"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e\") " Apr 23 18:59:27.930630 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930608 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:59:27.930789 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.930772 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:27.931187 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.931163 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-home" (OuterVolumeSpecName: "home") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:59:27.932907 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.932883 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-dshm" (OuterVolumeSpecName: "dshm") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:59:27.933007 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.932902 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 18:59:27.933007 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.932914 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kube-api-access-6bgnj" (OuterVolumeSpecName: "kube-api-access-6bgnj") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "kube-api-access-6bgnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 18:59:27.942400 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.942370 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:59:27.987316 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:27.987274 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" (UID: "5b3b04bc-90d0-46a2-a4b5-a17dadd7652e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 18:59:28.032275 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.032225 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:28.032275 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.032264 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:28.032275 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.032279 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:28.032511 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.032292 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bgnj\" (UniqueName: \"kubernetes.io/projected/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-kube-api-access-6bgnj\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:28.032511 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.032303 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:28.032511 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.032314 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 18:59:28.139653 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.139623 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh_5b3b04bc-90d0-46a2-a4b5-a17dadd7652e/main/0.log" Apr 23 18:59:28.139985 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.139962 2575 generic.go:358] "Generic (PLEG): container finished" podID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerID="7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d" exitCode=137 Apr 23 18:59:28.140062 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.140011 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" event={"ID":"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e","Type":"ContainerDied","Data":"7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d"} Apr 23 18:59:28.140062 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.140035 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" event={"ID":"5b3b04bc-90d0-46a2-a4b5-a17dadd7652e","Type":"ContainerDied","Data":"4380fd68976f8111edf342c9a49d02501bcf60b45061d1acd47b12b5d471f620"} Apr 23 18:59:28.140062 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.140039 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh" Apr 23 18:59:28.140062 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.140050 2575 scope.go:117] "RemoveContainer" containerID="7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d" Apr 23 18:59:28.148847 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.148818 2575 scope.go:117] "RemoveContainer" containerID="31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444" Apr 23 18:59:28.162741 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.162715 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh"] Apr 23 18:59:28.163046 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.163028 2575 scope.go:117] "RemoveContainer" containerID="7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d" Apr 23 18:59:28.163373 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:59:28.163354 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d\": container with ID starting with 7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d not found: ID does not exist" containerID="7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d" Apr 23 18:59:28.163468 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.163388 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d"} err="failed to get container status \"7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d\": rpc error: code = NotFound desc = could not find container \"7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d\": container with ID starting with 7262f726539110fc87d4ecfb8d303f0a98a869b9b7093cc9dd8c34f0b2ef2e2d not found: ID does not exist" Apr 23 18:59:28.163468 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.163415 2575 scope.go:117] "RemoveContainer" containerID="31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444" Apr 23 18:59:28.163730 ip-10-0-137-68 kubenswrapper[2575]: E0423 18:59:28.163710 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444\": container with ID starting with 31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444 not found: ID does not exist" containerID="31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444" Apr 23 18:59:28.163784 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.163740 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444"} err="failed to get container status \"31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444\": rpc error: code = NotFound desc = could not find container \"31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444\": container with ID starting with 31ef9eeab75abbf001928d776fe2e6db2f3cd4ef4769ec7ef49093a972e42444 not found: ID does not exist" Apr 23 18:59:28.166301 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:28.166269 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d75c5d878-d99kh"] Apr 23 18:59:29.595538 ip-10-0-137-68 kubenswrapper[2575]: I0423 18:59:29.595507 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" path="/var/lib/kubelet/pods/5b3b04bc-90d0-46a2-a4b5-a17dadd7652e/volumes" Apr 23 19:01:05.681080 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.680911 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht"] Apr 23 19:01:05.681565 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.681303 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" Apr 23 19:01:05.681565 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.681318 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" Apr 23 19:01:05.681565 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.681331 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="storage-initializer" Apr 23 19:01:05.681565 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.681336 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="storage-initializer" Apr 23 19:01:05.681565 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.681408 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b3b04bc-90d0-46a2-a4b5-a17dadd7652e" containerName="main" Apr 23 19:01:05.684502 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.684484 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.699854 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.699822 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\"" Apr 23 19:01:05.716219 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.716186 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht"] Apr 23 19:01:05.867785 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.867744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-home\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.867785 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.867790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-model-cache\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.868055 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.867822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-tmp-dir\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.868055 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.867903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b05bd-71c3-470d-a345-aafa59fe34d5-tls-certs\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.868055 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.867934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-dshm\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.868055 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.867954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2fz\" (UniqueName: \"kubernetes.io/projected/920b05bd-71c3-470d-a345-aafa59fe34d5-kube-api-access-2x2fz\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.868055 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.868019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969552 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969453 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-home\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969552 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969497 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-model-cache\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969552 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969539 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-tmp-dir\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969634 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b05bd-71c3-470d-a345-aafa59fe34d5-tls-certs\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969677 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-dshm\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969707 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2fz\" (UniqueName: \"kubernetes.io/projected/920b05bd-71c3-470d-a345-aafa59fe34d5-kube-api-access-2x2fz\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.969846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969745 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.970034 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.969978 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-home\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.970090 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.970024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-model-cache\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.970090 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.970043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-tmp-dir\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.970183 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.970149 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.971969 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.971945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-dshm\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.972277 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.972257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b05bd-71c3-470d-a345-aafa59fe34d5-tls-certs\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.981425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.981394 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2fz\" (UniqueName: \"kubernetes.io/projected/920b05bd-71c3-470d-a345-aafa59fe34d5-kube-api-access-2x2fz\") pod \"precise-prefix-cache-test-kserve-65747bccd4-kk5ht\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:05.995301 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:05.995274 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:06.135489 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:06.135461 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht"] Apr 23 19:01:06.138280 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:01:06.138248 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920b05bd_71c3_470d_a345_aafa59fe34d5.slice/crio-24474cfec66757e4efe568e4d1a1aad8d4457f7d4fef01636d8cf121d74e72c8 WatchSource:0}: Error finding container 24474cfec66757e4efe568e4d1a1aad8d4457f7d4fef01636d8cf121d74e72c8: Status 404 returned error can't find the container with id 24474cfec66757e4efe568e4d1a1aad8d4457f7d4fef01636d8cf121d74e72c8 Apr 23 19:01:06.475936 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:06.475895 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" event={"ID":"920b05bd-71c3-470d-a345-aafa59fe34d5","Type":"ContainerStarted","Data":"a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979"} Apr 23 19:01:06.476156 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:06.475943 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" event={"ID":"920b05bd-71c3-470d-a345-aafa59fe34d5","Type":"ContainerStarted","Data":"24474cfec66757e4efe568e4d1a1aad8d4457f7d4fef01636d8cf121d74e72c8"} Apr 23 19:01:10.491685 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:10.491646 2575 generic.go:358] "Generic (PLEG): container finished" podID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerID="a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979" exitCode=0 Apr 23 19:01:10.492177 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:10.491729 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" event={"ID":"920b05bd-71c3-470d-a345-aafa59fe34d5","Type":"ContainerDied","Data":"a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979"} Apr 23 19:01:11.496476 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:11.496440 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" event={"ID":"920b05bd-71c3-470d-a345-aafa59fe34d5","Type":"ContainerStarted","Data":"dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca"} Apr 23 19:01:11.516737 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:11.516688 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" podStartSLOduration=6.516670746 podStartE2EDuration="6.516670746s" podCreationTimestamp="2026-04-23 19:01:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:01:11.515322922 +0000 UTC m=+4770.534775961" watchObservedRunningTime="2026-04-23 19:01:11.516670746 +0000 UTC m=+4770.536123762" Apr 23 19:01:15.995607 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:15.995551 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:15.995607 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:15.995614 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:16.008527 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:16.008492 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:16.527009 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:16.526979 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:27.556663 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:27.556618 2575 generic.go:358] "Generic (PLEG): container finished" podID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerID="fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef" exitCode=0 Apr 23 19:01:27.557098 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:27.556687 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" event={"ID":"7b1bae63-62aa-43d7-b009-a2b70953fd84","Type":"ContainerDied","Data":"fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef"} Apr 23 19:01:28.561986 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:28.561942 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" event={"ID":"7b1bae63-62aa-43d7-b009-a2b70953fd84","Type":"ContainerStarted","Data":"5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3"} Apr 23 19:01:28.583696 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:28.583630 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podStartSLOduration=135.583611367 podStartE2EDuration="2m15.583611367s" podCreationTimestamp="2026-04-23 18:59:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:01:28.581470318 +0000 UTC m=+4787.600923334" watchObservedRunningTime="2026-04-23 19:01:28.583611367 +0000 UTC m=+4787.603064375" Apr 23 19:01:34.068306 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:34.068264 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 19:01:34.068306 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:34.068314 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 19:01:34.069944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:34.069915 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:01:38.305409 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.305369 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht"] Apr 23 19:01:38.305946 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.305789 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerName="main" containerID="cri-o://dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca" gracePeriod=30 Apr 23 19:01:38.580720 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.580694 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:38.602934 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.602896 2575 generic.go:358] "Generic (PLEG): container finished" podID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerID="dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca" exitCode=0 Apr 23 19:01:38.603121 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.602990 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" Apr 23 19:01:38.603121 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.603002 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" event={"ID":"920b05bd-71c3-470d-a345-aafa59fe34d5","Type":"ContainerDied","Data":"dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca"} Apr 23 19:01:38.603121 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.603042 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht" event={"ID":"920b05bd-71c3-470d-a345-aafa59fe34d5","Type":"ContainerDied","Data":"24474cfec66757e4efe568e4d1a1aad8d4457f7d4fef01636d8cf121d74e72c8"} Apr 23 19:01:38.603121 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.603063 2575 scope.go:117] "RemoveContainer" containerID="dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca" Apr 23 19:01:38.612416 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.612397 2575 scope.go:117] "RemoveContainer" containerID="a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979" Apr 23 19:01:38.660755 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660717 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-dshm\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.660755 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660763 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-home\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.661020 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660819 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x2fz\" (UniqueName: \"kubernetes.io/projected/920b05bd-71c3-470d-a345-aafa59fe34d5-kube-api-access-2x2fz\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.661020 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660852 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-model-cache\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.661020 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660897 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-tmp-dir\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.661020 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660936 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b05bd-71c3-470d-a345-aafa59fe34d5-tls-certs\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.661020 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.660960 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-kserve-provision-location\") pod \"920b05bd-71c3-470d-a345-aafa59fe34d5\" (UID: \"920b05bd-71c3-470d-a345-aafa59fe34d5\") " Apr 23 19:01:38.661309 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.661239 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-home" (OuterVolumeSpecName: "home") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:01:38.661394 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.661316 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:01:38.661394 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.661385 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:01:38.663324 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.663280 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920b05bd-71c3-470d-a345-aafa59fe34d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:01:38.663459 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.663435 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920b05bd-71c3-470d-a345-aafa59fe34d5-kube-api-access-2x2fz" (OuterVolumeSpecName: "kube-api-access-2x2fz") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "kube-api-access-2x2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:01:38.663545 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.663482 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-dshm" (OuterVolumeSpecName: "dshm") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:01:38.680208 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.680118 2575 scope.go:117] "RemoveContainer" containerID="dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca" Apr 23 19:01:38.680499 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:01:38.680470 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca\": container with ID starting with dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca not found: ID does not exist" containerID="dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca" Apr 23 19:01:38.680564 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.680511 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca"} err="failed to get container status \"dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca\": rpc error: code = NotFound desc = could not find container \"dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca\": container with ID starting with dc02a79563900504c97dfcc826fa821282064440c0b65702f56f1e61a7e9fbca not found: ID does not exist" Apr 23 19:01:38.680564 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.680533 2575 scope.go:117] "RemoveContainer" containerID="a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979" Apr 23 19:01:38.680899 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:01:38.680873 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979\": container with ID starting with a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979 not found: ID does not exist" containerID="a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979" Apr 23 19:01:38.680971 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.680910 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979"} err="failed to get container status \"a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979\": rpc error: code = NotFound desc = could not find container \"a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979\": container with ID starting with a1b5876f6ed3f2559d3b2d1a367e36f2b7083615600b26fcfc9a8969b28df979 not found: ID does not exist" Apr 23 19:01:38.727704 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.727655 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "920b05bd-71c3-470d-a345-aafa59fe34d5" (UID: "920b05bd-71c3-470d-a345-aafa59fe34d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:01:38.762348 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762307 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.762348 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762345 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.762543 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762358 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2fz\" (UniqueName: \"kubernetes.io/projected/920b05bd-71c3-470d-a345-aafa59fe34d5-kube-api-access-2x2fz\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.762543 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762371 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.762543 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762385 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.762543 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762396 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b05bd-71c3-470d-a345-aafa59fe34d5-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.762543 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.762407 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920b05bd-71c3-470d-a345-aafa59fe34d5-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:01:38.925516 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.925479 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht"] Apr 23 19:01:38.931103 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:38.931062 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-65747bccd4-kk5ht"] Apr 23 19:01:39.596389 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:39.596346 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" path="/var/lib/kubelet/pods/920b05bd-71c3-470d-a345-aafa59fe34d5/volumes" Apr 23 19:01:41.773147 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:41.773024 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:01:41.780443 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:41.780415 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:01:44.068518 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:44.068462 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:01:49.638777 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.638738 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg"] Apr 23 19:01:49.639184 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.639142 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerName="main" Apr 23 19:01:49.639184 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.639157 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerName="main" Apr 23 19:01:49.639184 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.639172 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerName="storage-initializer" Apr 23 19:01:49.639184 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.639181 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerName="storage-initializer" Apr 23 19:01:49.639422 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.639318 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="920b05bd-71c3-470d-a345-aafa59fe34d5" containerName="main" Apr 23 19:01:49.645056 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.645027 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.647367 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.647341 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\"" Apr 23 19:01:49.653627 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.653358 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg"] Apr 23 19:01:49.764366 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-model-cache\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.764563 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcst\" (UniqueName: \"kubernetes.io/projected/850763ef-820f-4f59-a8d9-3b0dc7307120-kube-api-access-wpcst\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.764563 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850763ef-820f-4f59-a8d9-3b0dc7307120-tls-certs\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.764563 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-kserve-provision-location\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.764563 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-dshm\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.764563 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-tmp-dir\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.764808 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.764578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-home\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865226 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865189 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-model-cache\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865421 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865248 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcst\" (UniqueName: \"kubernetes.io/projected/850763ef-820f-4f59-a8d9-3b0dc7307120-kube-api-access-wpcst\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865421 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865278 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850763ef-820f-4f59-a8d9-3b0dc7307120-tls-certs\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865421 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865295 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-kserve-provision-location\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865421 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865310 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-dshm\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865421 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865334 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-tmp-dir\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865729 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865420 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-home\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865729 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-model-cache\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865729 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-kserve-provision-location\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.865979 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.865954 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-tmp-dir\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.866105 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.866008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-home\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.867755 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.867734 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-dshm\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.868133 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.868111 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850763ef-820f-4f59-a8d9-3b0dc7307120-tls-certs\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.873957 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.873935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcst\" (UniqueName: \"kubernetes.io/projected/850763ef-820f-4f59-a8d9-3b0dc7307120-kube-api-access-wpcst\") pod \"stop-feature-test-kserve-b8c47cf74-d8fqg\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:49.958287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:49.958193 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:50.303414 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:50.303381 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg"] Apr 23 19:01:50.304991 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:01:50.304964 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850763ef_820f_4f59_a8d9_3b0dc7307120.slice/crio-8918424bfb7d6f2128e77889f7a9a68809398d9fac347c79d9fde66ecb6220e4 WatchSource:0}: Error finding container 8918424bfb7d6f2128e77889f7a9a68809398d9fac347c79d9fde66ecb6220e4: Status 404 returned error can't find the container with id 8918424bfb7d6f2128e77889f7a9a68809398d9fac347c79d9fde66ecb6220e4 Apr 23 19:01:50.649370 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:50.649270 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" event={"ID":"850763ef-820f-4f59-a8d9-3b0dc7307120","Type":"ContainerStarted","Data":"bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246"} Apr 23 19:01:50.649370 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:50.649318 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" event={"ID":"850763ef-820f-4f59-a8d9-3b0dc7307120","Type":"ContainerStarted","Data":"8918424bfb7d6f2128e77889f7a9a68809398d9fac347c79d9fde66ecb6220e4"} Apr 23 19:01:54.068464 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:54.068401 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:01:54.666909 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:54.666804 2575 generic.go:358] "Generic (PLEG): container finished" podID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerID="bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246" exitCode=0 Apr 23 19:01:54.666909 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:54.666879 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" event={"ID":"850763ef-820f-4f59-a8d9-3b0dc7307120","Type":"ContainerDied","Data":"bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246"} Apr 23 19:01:55.672522 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:55.672482 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" event={"ID":"850763ef-820f-4f59-a8d9-3b0dc7307120","Type":"ContainerStarted","Data":"93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77"} Apr 23 19:01:55.694889 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:55.694833 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podStartSLOduration=6.694817183 podStartE2EDuration="6.694817183s" podCreationTimestamp="2026-04-23 19:01:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:01:55.693852634 +0000 UTC m=+4814.713305662" watchObservedRunningTime="2026-04-23 19:01:55.694817183 +0000 UTC m=+4814.714270199" Apr 23 19:01:59.958606 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:59.958551 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:59.958606 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:59.958619 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:01:59.960303 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:01:59.960263 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:02:04.069137 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:04.069087 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:02:09.959135 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:09.959077 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:02:14.068242 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:14.068195 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:02:19.958966 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:19.958909 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:02:24.069121 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:24.069018 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:02:29.958947 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:29.958897 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:02:34.069029 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:34.068974 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:02:39.958907 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:39.958853 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:02:44.068848 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:44.068792 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:02:49.958969 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:49.958920 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:02:54.069015 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:54.068965 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:02:59.959493 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:02:59.959443 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:03:04.069119 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:04.069075 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:03:09.959452 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:09.959405 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:03:14.069032 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:14.068985 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Apr 23 19:03:19.959584 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:19.959541 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:03:24.078268 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:24.078234 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 19:03:24.086306 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:24.086280 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 19:03:29.958750 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:29.958704 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:03:32.879405 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:32.879357 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2"] Apr 23 19:03:32.879834 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:32.879757 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" containerID="cri-o://5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3" gracePeriod=30 Apr 23 19:03:39.959661 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:39.959611 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Apr 23 19:03:44.265368 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.265330 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv"] Apr 23 19:03:44.270986 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.270962 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.273485 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.273456 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\"" Apr 23 19:03:44.281100 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.281075 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv"] Apr 23 19:03:44.331332 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjjj\" (UniqueName: \"kubernetes.io/projected/9e541372-1023-4b32-b170-8b252dfd62f2-kube-api-access-4zjjj\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.331515 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e541372-1023-4b32-b170-8b252dfd62f2-tls-certs\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.331515 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-dshm\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.331515 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-model-cache\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.331515 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-tmp-dir\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.331515 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-kserve-provision-location\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.331857 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.331518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-home\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.432950 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.432913 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-dshm\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.432950 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.432952 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-model-cache\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433210 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.432971 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-tmp-dir\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433210 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433120 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-kserve-provision-location\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433210 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433179 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-home\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433379 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433252 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjjj\" (UniqueName: \"kubernetes.io/projected/9e541372-1023-4b32-b170-8b252dfd62f2-kube-api-access-4zjjj\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433379 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-tmp-dir\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433379 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433365 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e541372-1023-4b32-b170-8b252dfd62f2-tls-certs\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433524 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433421 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-model-cache\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433524 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-kserve-provision-location\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.433672 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.433550 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-home\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.435285 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.435262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-dshm\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.435823 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.435802 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e541372-1023-4b32-b170-8b252dfd62f2-tls-certs\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.442216 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.442189 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjjj\" (UniqueName: \"kubernetes.io/projected/9e541372-1023-4b32-b170-8b252dfd62f2-kube-api-access-4zjjj\") pod \"router-with-refs-test-kserve-c74cb6db6-m7zsv\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.582988 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.582908 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:03:44.919816 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:44.919774 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv"] Apr 23 19:03:44.921816 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:03:44.921784 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e541372_1023_4b32_b170_8b252dfd62f2.slice/crio-839bfa9568dd240f9c2542635c68a08570a4d330a29b3d9d2d8d19b092bdca3b WatchSource:0}: Error finding container 839bfa9568dd240f9c2542635c68a08570a4d330a29b3d9d2d8d19b092bdca3b: Status 404 returned error can't find the container with id 839bfa9568dd240f9c2542635c68a08570a4d330a29b3d9d2d8d19b092bdca3b Apr 23 19:03:45.091309 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:45.091264 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" event={"ID":"9e541372-1023-4b32-b170-8b252dfd62f2","Type":"ContainerStarted","Data":"7c996a7f78d1e3b7c51482e9dd1a1e016ce91b1cf6f9365b66169059d7486136"} Apr 23 19:03:45.091309 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:45.091312 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" event={"ID":"9e541372-1023-4b32-b170-8b252dfd62f2","Type":"ContainerStarted","Data":"839bfa9568dd240f9c2542635c68a08570a4d330a29b3d9d2d8d19b092bdca3b"} Apr 23 19:03:49.969791 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:49.969758 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:03:49.978000 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:03:49.977967 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:04:03.145521 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.145458 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-7cc767b4f4-wpnr2_7b1bae63-62aa-43d7-b009-a2b70953fd84/main/0.log" Apr 23 19:04:03.145874 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.145859 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 19:04:03.153399 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.153377 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-7cc767b4f4-wpnr2_7b1bae63-62aa-43d7-b009-a2b70953fd84/main/0.log" Apr 23 19:04:03.153767 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.153743 2575 generic.go:358] "Generic (PLEG): container finished" podID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerID="5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3" exitCode=137 Apr 23 19:04:03.153843 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.153813 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" Apr 23 19:04:03.153882 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.153841 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" event={"ID":"7b1bae63-62aa-43d7-b009-a2b70953fd84","Type":"ContainerDied","Data":"5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3"} Apr 23 19:04:03.153916 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.153881 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2" event={"ID":"7b1bae63-62aa-43d7-b009-a2b70953fd84","Type":"ContainerDied","Data":"be46efa6f20e32e2810bada890981443dd67c35db408aabbb75033c01a3c9033"} Apr 23 19:04:03.153916 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.153898 2575 scope.go:117] "RemoveContainer" containerID="5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3" Apr 23 19:04:03.161434 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.161418 2575 scope.go:117] "RemoveContainer" containerID="fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef" Apr 23 19:04:03.223403 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.223379 2575 scope.go:117] "RemoveContainer" containerID="5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3" Apr 23 19:04:03.223736 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:04:03.223713 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3\": container with ID starting with 5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3 not found: ID does not exist" containerID="5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3" Apr 23 19:04:03.223838 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.223750 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3"} err="failed to get container status \"5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3\": rpc error: code = NotFound desc = could not find container \"5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3\": container with ID starting with 5312ba9927da303e709cf09dfe96e50b07bc8507761c6349dcea979e16c0c9e3 not found: ID does not exist" Apr 23 19:04:03.223838 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.223778 2575 scope.go:117] "RemoveContainer" containerID="fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef" Apr 23 19:04:03.224071 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:04:03.224044 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef\": container with ID starting with fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef not found: ID does not exist" containerID="fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef" Apr 23 19:04:03.224122 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.224069 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef"} err="failed to get container status \"fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef\": rpc error: code = NotFound desc = could not find container \"fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef\": container with ID starting with fd7c90cae4882ac56cda2268b018679dbc8a0d7feffc929e7d097cc2dd8f1fef not found: ID does not exist" Apr 23 19:04:03.304257 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304223 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-kserve-provision-location\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304274 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr7gx\" (UniqueName: \"kubernetes.io/projected/7b1bae63-62aa-43d7-b009-a2b70953fd84-kube-api-access-kr7gx\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304295 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-model-cache\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304315 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-home\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304373 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1bae63-62aa-43d7-b009-a2b70953fd84-tls-certs\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304661 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304423 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-dshm\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304661 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304446 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-tmp-dir\") pod \"7b1bae63-62aa-43d7-b009-a2b70953fd84\" (UID: \"7b1bae63-62aa-43d7-b009-a2b70953fd84\") " Apr 23 19:04:03.304661 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304540 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:03.304847 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.304817 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.305320 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.305284 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-home" (OuterVolumeSpecName: "home") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:03.306647 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.306582 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-dshm" (OuterVolumeSpecName: "dshm") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:03.306743 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.306701 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1bae63-62aa-43d7-b009-a2b70953fd84-kube-api-access-kr7gx" (OuterVolumeSpecName: "kube-api-access-kr7gx") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "kube-api-access-kr7gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:04:03.306822 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.306799 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1bae63-62aa-43d7-b009-a2b70953fd84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:04:03.316555 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.316532 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:03.360558 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.360529 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b1bae63-62aa-43d7-b009-a2b70953fd84" (UID: "7b1bae63-62aa-43d7-b009-a2b70953fd84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:03.405397 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.405341 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7gx\" (UniqueName: \"kubernetes.io/projected/7b1bae63-62aa-43d7-b009-a2b70953fd84-kube-api-access-kr7gx\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.405397 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.405362 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.405397 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.405374 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1bae63-62aa-43d7-b009-a2b70953fd84-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.405397 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.405382 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.405397 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.405390 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.405397 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.405398 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bae63-62aa-43d7-b009-a2b70953fd84-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:03.476762 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.476736 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2"] Apr 23 19:04:03.480923 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.480896 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7cc767b4f4-wpnr2"] Apr 23 19:04:03.595150 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:03.595109 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" path="/var/lib/kubelet/pods/7b1bae63-62aa-43d7-b009-a2b70953fd84/volumes" Apr 23 19:04:12.724908 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:12.724870 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg"] Apr 23 19:04:12.725355 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:12.725153 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" containerID="cri-o://93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77" gracePeriod=30 Apr 23 19:04:30.251774 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:30.251737 2575 generic.go:358] "Generic (PLEG): container finished" podID="9e541372-1023-4b32-b170-8b252dfd62f2" containerID="7c996a7f78d1e3b7c51482e9dd1a1e016ce91b1cf6f9365b66169059d7486136" exitCode=0 Apr 23 19:04:30.252203 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:30.251806 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" event={"ID":"9e541372-1023-4b32-b170-8b252dfd62f2","Type":"ContainerDied","Data":"7c996a7f78d1e3b7c51482e9dd1a1e016ce91b1cf6f9365b66169059d7486136"} Apr 23 19:04:30.252901 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:30.252882 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 19:04:31.256720 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:31.256689 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" event={"ID":"9e541372-1023-4b32-b170-8b252dfd62f2","Type":"ContainerStarted","Data":"981e84c39b12d15277f47f41adc6853ff61d343622f99e611b3a2e635d5a7577"} Apr 23 19:04:31.295016 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:31.294956 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podStartSLOduration=47.294934833 podStartE2EDuration="47.294934833s" podCreationTimestamp="2026-04-23 19:03:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:04:31.293827883 +0000 UTC m=+4970.313280899" watchObservedRunningTime="2026-04-23 19:04:31.294934833 +0000 UTC m=+4970.314387852" Apr 23 19:04:34.583343 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:34.583304 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:04:34.583868 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:34.583721 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:04:34.584894 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:34.584857 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:04:42.994945 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:42.994917 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-b8c47cf74-d8fqg_850763ef-820f-4f59-a8d9-3b0dc7307120/main/0.log" Apr 23 19:04:42.995356 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:42.995339 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:04:43.125070 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125035 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpcst\" (UniqueName: \"kubernetes.io/projected/850763ef-820f-4f59-a8d9-3b0dc7307120-kube-api-access-wpcst\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125070 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125079 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-dshm\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125333 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125112 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-model-cache\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125333 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125126 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-tmp-dir\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125333 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125150 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-home\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125333 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125233 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850763ef-820f-4f59-a8d9-3b0dc7307120-tls-certs\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125333 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125259 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-kserve-provision-location\") pod \"850763ef-820f-4f59-a8d9-3b0dc7307120\" (UID: \"850763ef-820f-4f59-a8d9-3b0dc7307120\") " Apr 23 19:04:43.125615 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125371 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-model-cache" (OuterVolumeSpecName: "model-cache") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:43.125615 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.125528 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.126037 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.126005 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-home" (OuterVolumeSpecName: "home") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:43.127399 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.127373 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850763ef-820f-4f59-a8d9-3b0dc7307120-kube-api-access-wpcst" (OuterVolumeSpecName: "kube-api-access-wpcst") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "kube-api-access-wpcst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:04:43.127511 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.127492 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850763ef-820f-4f59-a8d9-3b0dc7307120-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:04:43.127709 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.127689 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-dshm" (OuterVolumeSpecName: "dshm") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:43.137316 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.137280 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:43.194469 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.194428 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "850763ef-820f-4f59-a8d9-3b0dc7307120" (UID: "850763ef-820f-4f59-a8d9-3b0dc7307120"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:04:43.226724 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.226683 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850763ef-820f-4f59-a8d9-3b0dc7307120-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.226724 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.226718 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.226724 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.226729 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpcst\" (UniqueName: \"kubernetes.io/projected/850763ef-820f-4f59-a8d9-3b0dc7307120-kube-api-access-wpcst\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.226961 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.226738 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.226961 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.226749 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.226961 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.226756 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850763ef-820f-4f59-a8d9-3b0dc7307120-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:04:43.306262 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.306171 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-b8c47cf74-d8fqg_850763ef-820f-4f59-a8d9-3b0dc7307120/main/0.log" Apr 23 19:04:43.306600 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.306554 2575 generic.go:358] "Generic (PLEG): container finished" podID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerID="93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77" exitCode=137 Apr 23 19:04:43.306732 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.306611 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" event={"ID":"850763ef-820f-4f59-a8d9-3b0dc7307120","Type":"ContainerDied","Data":"93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77"} Apr 23 19:04:43.306732 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.306643 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" Apr 23 19:04:43.306732 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.306657 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg" event={"ID":"850763ef-820f-4f59-a8d9-3b0dc7307120","Type":"ContainerDied","Data":"8918424bfb7d6f2128e77889f7a9a68809398d9fac347c79d9fde66ecb6220e4"} Apr 23 19:04:43.306732 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.306674 2575 scope.go:117] "RemoveContainer" containerID="93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77" Apr 23 19:04:43.315709 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.315691 2575 scope.go:117] "RemoveContainer" containerID="bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246" Apr 23 19:04:43.326394 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.326372 2575 scope.go:117] "RemoveContainer" containerID="93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77" Apr 23 19:04:43.326721 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:04:43.326690 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77\": container with ID starting with 93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77 not found: ID does not exist" containerID="93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77" Apr 23 19:04:43.326825 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.326733 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77"} err="failed to get container status \"93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77\": rpc error: code = NotFound desc = could not find container \"93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77\": container with ID starting with 93d053f3b7dbd5d3f1a6253160d7d2f456a636a3009f36ca63a47e05c736ce77 not found: ID does not exist" Apr 23 19:04:43.326825 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.326762 2575 scope.go:117] "RemoveContainer" containerID="bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246" Apr 23 19:04:43.327140 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:04:43.327120 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246\": container with ID starting with bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246 not found: ID does not exist" containerID="bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246" Apr 23 19:04:43.327230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.327146 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246"} err="failed to get container status \"bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246\": rpc error: code = NotFound desc = could not find container \"bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246\": container with ID starting with bd53be191cdc8c4e33e206b66f7f6196253ade846cd1ed9211b4fc8307f39246 not found: ID does not exist" Apr 23 19:04:43.329998 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.329967 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg"] Apr 23 19:04:43.336174 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.336143 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-d8fqg"] Apr 23 19:04:43.595816 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:43.595722 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" path="/var/lib/kubelet/pods/850763ef-820f-4f59-a8d9-3b0dc7307120/volumes" Apr 23 19:04:44.583429 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:44.583370 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:04:46.168853 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.168804 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt"] Apr 23 19:04:46.169372 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169319 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" Apr 23 19:04:46.169372 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169342 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" Apr 23 19:04:46.169372 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169370 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="storage-initializer" Apr 23 19:04:46.169525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169379 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="storage-initializer" Apr 23 19:04:46.169525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169399 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="storage-initializer" Apr 23 19:04:46.169525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169408 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="storage-initializer" Apr 23 19:04:46.169525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169423 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" Apr 23 19:04:46.169525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169431 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" Apr 23 19:04:46.169525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169510 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b1bae63-62aa-43d7-b009-a2b70953fd84" containerName="main" Apr 23 19:04:46.169844 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.169538 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="850763ef-820f-4f59-a8d9-3b0dc7307120" containerName="main" Apr 23 19:04:46.210208 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.210166 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt"] Apr 23 19:04:46.210398 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.210251 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.212620 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.212579 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\"" Apr 23 19:04:46.256891 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.256849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-tmp-dir\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.257101 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.256913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-model-cache\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.257101 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.256977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-dshm\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.257101 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.257017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a546aa0-a49f-4b34-8326-080fd2c3c743-tls-certs\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.257101 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.257044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcbf\" (UniqueName: \"kubernetes.io/projected/1a546aa0-a49f-4b34-8326-080fd2c3c743-kube-api-access-9wcbf\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.257273 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.257111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-kserve-provision-location\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.257273 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.257196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-home\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.358669 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.358627 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-tmp-dir\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.358872 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.358684 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-model-cache\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.358872 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.358706 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-dshm\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.358872 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.358727 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a546aa0-a49f-4b34-8326-080fd2c3c743-tls-certs\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.358872 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.358753 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcbf\" (UniqueName: \"kubernetes.io/projected/1a546aa0-a49f-4b34-8326-080fd2c3c743-kube-api-access-9wcbf\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.358872 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.358793 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-kserve-provision-location\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.359120 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.359095 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-home\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.359175 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.359141 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-tmp-dir\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.359229 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.359191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-model-cache\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.359318 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.359295 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-kserve-provision-location\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.359490 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.359465 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-home\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.361069 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.361045 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-dshm\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.361423 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.361404 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a546aa0-a49f-4b34-8326-080fd2c3c743-tls-certs\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.369697 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.369665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcbf\" (UniqueName: \"kubernetes.io/projected/1a546aa0-a49f-4b34-8326-080fd2c3c743-kube-api-access-9wcbf\") pod \"stop-feature-test-kserve-b8c47cf74-mvwrt\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.521527 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.521481 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:46.668681 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:46.668557 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt"] Apr 23 19:04:46.671765 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:04:46.671733 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a546aa0_a49f_4b34_8326_080fd2c3c743.slice/crio-3ff2fa315ef313e3690b6a7fbefcf6e208e9d223258bd7520852049081b2061e WatchSource:0}: Error finding container 3ff2fa315ef313e3690b6a7fbefcf6e208e9d223258bd7520852049081b2061e: Status 404 returned error can't find the container with id 3ff2fa315ef313e3690b6a7fbefcf6e208e9d223258bd7520852049081b2061e Apr 23 19:04:47.322913 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:47.322876 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" event={"ID":"1a546aa0-a49f-4b34-8326-080fd2c3c743","Type":"ContainerStarted","Data":"06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5"} Apr 23 19:04:47.322913 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:47.322916 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" event={"ID":"1a546aa0-a49f-4b34-8326-080fd2c3c743","Type":"ContainerStarted","Data":"3ff2fa315ef313e3690b6a7fbefcf6e208e9d223258bd7520852049081b2061e"} Apr 23 19:04:51.340753 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:51.340716 2575 generic.go:358] "Generic (PLEG): container finished" podID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerID="06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5" exitCode=0 Apr 23 19:04:51.341132 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:51.340783 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" event={"ID":"1a546aa0-a49f-4b34-8326-080fd2c3c743","Type":"ContainerDied","Data":"06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5"} Apr 23 19:04:52.348926 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:52.348883 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" event={"ID":"1a546aa0-a49f-4b34-8326-080fd2c3c743","Type":"ContainerStarted","Data":"1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247"} Apr 23 19:04:52.374058 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:52.374006 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podStartSLOduration=6.373988552 podStartE2EDuration="6.373988552s" podCreationTimestamp="2026-04-23 19:04:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:04:52.372717761 +0000 UTC m=+4991.392170777" watchObservedRunningTime="2026-04-23 19:04:52.373988552 +0000 UTC m=+4991.393441568" Apr 23 19:04:54.584153 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:54.584095 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:04:56.522558 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:56.522518 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:56.522558 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:56.522562 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:04:56.524119 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:04:56.524083 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:05:04.584018 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:04.583962 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:05:06.522528 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:06.522475 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:05:14.583446 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:14.583399 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:05:16.522758 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:16.522712 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:05:24.584103 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:24.584010 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:05:26.522304 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:26.522255 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:05:34.583539 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:34.583484 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:05:36.522734 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:36.522682 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:05:44.583361 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:44.583315 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:05:46.521930 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:46.521883 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:05:54.584049 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:54.584011 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:05:56.522065 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:05:56.522012 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:06:04.584205 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:04.584150 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:06:06.522290 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:06.522241 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:06:14.584340 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:14.584271 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused" Apr 23 19:06:16.522624 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:16.522553 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:06:24.593883 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:24.593850 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:06:24.601649 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:24.601620 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:06:26.522244 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:26.522199 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:06:33.559549 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:33.559512 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv"] Apr 23 19:06:33.560157 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:33.559815 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" containerID="cri-o://981e84c39b12d15277f47f41adc6853ff61d343622f99e611b3a2e635d5a7577" gracePeriod=30 Apr 23 19:06:36.522465 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:36.522423 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:06:41.799711 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:41.799553 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:06:41.807628 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:41.807603 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:06:46.521938 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:46.521894 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Apr 23 19:06:48.074235 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.074195 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9"] Apr 23 19:06:48.079550 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.079527 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.083956 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.083925 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-jk4zp\"" Apr 23 19:06:48.084569 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.084547 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\"" Apr 23 19:06:48.103525 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.103490 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9"] Apr 23 19:06:48.171078 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.171281 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367a7f17-7395-4728-b2ff-486dcae24265-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.171281 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.171281 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.171281 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.171281 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.171551 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.171375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8vz\" (UniqueName: \"kubernetes.io/projected/367a7f17-7395-4728-b2ff-486dcae24265-kube-api-access-pw8vz\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.188162 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.188122 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr"] Apr 23 19:06:48.192696 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.192670 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.215946 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.215911 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr"] Apr 23 19:06:48.272470 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.272684 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272496 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8vz\" (UniqueName: \"kubernetes.io/projected/367a7f17-7395-4728-b2ff-486dcae24265-kube-api-access-pw8vz\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.272684 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.272684 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.272684 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.272859 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272700 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.272859 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9259e06d-020f-4a51-a5e4-f116d363a173-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.272859 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfjp\" (UniqueName: \"kubernetes.io/projected/9259e06d-020f-4a51-a5e4-f116d363a173-kube-api-access-6bfjp\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.272859 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272785 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367a7f17-7395-4728-b2ff-486dcae24265-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.272859 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272812 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.272859 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272844 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.273109 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272865 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.273109 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.273109 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.272953 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.273109 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.273095 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.273293 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.273235 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.273293 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.273257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.273378 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.273290 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.275175 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.275144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.275340 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.275323 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367a7f17-7395-4728-b2ff-486dcae24265-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.284949 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.284919 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8vz\" (UniqueName: \"kubernetes.io/projected/367a7f17-7395-4728-b2ff-486dcae24265-kube-api-access-pw8vz\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.374259 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374149 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9259e06d-020f-4a51-a5e4-f116d363a173-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374259 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374197 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfjp\" (UniqueName: \"kubernetes.io/projected/9259e06d-020f-4a51-a5e4-f116d363a173-kube-api-access-6bfjp\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374259 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374231 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374548 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374354 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374548 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374429 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374548 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374463 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374548 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374483 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374780 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374780 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374878 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374803 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.374878 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.374859 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.376716 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.376698 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.376902 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.376885 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9259e06d-020f-4a51-a5e4-f116d363a173-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.382646 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.382622 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfjp\" (UniqueName: \"kubernetes.io/projected/9259e06d-020f-4a51-a5e4-f116d363a173-kube-api-access-6bfjp\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.389613 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.389571 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:48.503279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.503233 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:48.531731 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.531700 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9"] Apr 23 19:06:48.534252 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:06:48.534211 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367a7f17_7395_4728_b2ff_486dcae24265.slice/crio-5c0d400074f3962fb706251cd95a08eb905ce904bd32c8f28b19007ff7a06e25 WatchSource:0}: Error finding container 5c0d400074f3962fb706251cd95a08eb905ce904bd32c8f28b19007ff7a06e25: Status 404 returned error can't find the container with id 5c0d400074f3962fb706251cd95a08eb905ce904bd32c8f28b19007ff7a06e25 Apr 23 19:06:48.643182 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.643152 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr"] Apr 23 19:06:48.645368 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:06:48.645333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9259e06d_020f_4a51_a5e4_f116d363a173.slice/crio-6da5382f6e95eb6c303738dad5bfc15b1cd7ed012e133f2102f28601857c1c72 WatchSource:0}: Error finding container 6da5382f6e95eb6c303738dad5bfc15b1cd7ed012e133f2102f28601857c1c72: Status 404 returned error can't find the container with id 6da5382f6e95eb6c303738dad5bfc15b1cd7ed012e133f2102f28601857c1c72 Apr 23 19:06:48.786884 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.786839 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerStarted","Data":"5c0d400074f3962fb706251cd95a08eb905ce904bd32c8f28b19007ff7a06e25"} Apr 23 19:06:48.788417 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.788390 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" event={"ID":"9259e06d-020f-4a51-a5e4-f116d363a173","Type":"ContainerStarted","Data":"481040fb03b31edb59876c30979e8e7e0ec11eee24c4202cc6ccbd1b2441de0f"} Apr 23 19:06:48.788556 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:48.788423 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" event={"ID":"9259e06d-020f-4a51-a5e4-f116d363a173","Type":"ContainerStarted","Data":"6da5382f6e95eb6c303738dad5bfc15b1cd7ed012e133f2102f28601857c1c72"} Apr 23 19:06:49.796511 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:49.796469 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerStarted","Data":"4f752c18d541fce3d8709cdf5125823a4b04bd997c313ad32a871ee331735382"} Apr 23 19:06:49.796981 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:49.796790 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:50.802338 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:50.802298 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerStarted","Data":"d64a193588831842be654fe80110505a0f854f7ae45ea376c0b60e989ce8504c"} Apr 23 19:06:53.815786 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:53.815736 2575 generic.go:358] "Generic (PLEG): container finished" podID="9259e06d-020f-4a51-a5e4-f116d363a173" containerID="481040fb03b31edb59876c30979e8e7e0ec11eee24c4202cc6ccbd1b2441de0f" exitCode=0 Apr 23 19:06:53.816190 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:53.815813 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" event={"ID":"9259e06d-020f-4a51-a5e4-f116d363a173","Type":"ContainerDied","Data":"481040fb03b31edb59876c30979e8e7e0ec11eee24c4202cc6ccbd1b2441de0f"} Apr 23 19:06:54.820384 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:54.820340 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" event={"ID":"9259e06d-020f-4a51-a5e4-f116d363a173","Type":"ContainerStarted","Data":"fd04574716781999cc2cae77415a7bd36fe2015b7106a3006cee9f787512bdf4"} Apr 23 19:06:54.822027 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:54.821995 2575 generic.go:358] "Generic (PLEG): container finished" podID="367a7f17-7395-4728-b2ff-486dcae24265" containerID="d64a193588831842be654fe80110505a0f854f7ae45ea376c0b60e989ce8504c" exitCode=0 Apr 23 19:06:54.822146 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:54.822065 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerDied","Data":"d64a193588831842be654fe80110505a0f854f7ae45ea376c0b60e989ce8504c"} Apr 23 19:06:54.844662 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:54.844572 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podStartSLOduration=6.844553572 podStartE2EDuration="6.844553572s" podCreationTimestamp="2026-04-23 19:06:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:06:54.843080209 +0000 UTC m=+5113.862533224" watchObservedRunningTime="2026-04-23 19:06:54.844553572 +0000 UTC m=+5113.864006589" Apr 23 19:06:55.827630 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:55.827575 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerStarted","Data":"c013baf683166c86306d551670d0cddd8208ed261cf00d241180f5d301bbf867"} Apr 23 19:06:55.851890 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:55.851829 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podStartSLOduration=7.03715868 podStartE2EDuration="7.85180894s" podCreationTimestamp="2026-04-23 19:06:48 +0000 UTC" firstStartedPulling="2026-04-23 19:06:48.53620642 +0000 UTC m=+5107.555659413" lastFinishedPulling="2026-04-23 19:06:49.35085668 +0000 UTC m=+5108.370309673" observedRunningTime="2026-04-23 19:06:55.849152982 +0000 UTC m=+5114.868605999" watchObservedRunningTime="2026-04-23 19:06:55.85180894 +0000 UTC m=+5114.871261957" Apr 23 19:06:56.532371 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:56.532336 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:06:56.541256 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:56.541220 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:06:58.389873 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:58.389839 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:58.390388 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:58.389884 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:06:58.391643 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:58.391609 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:06:58.503857 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:58.503818 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:58.504075 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:58.503868 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:06:58.505681 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:06:58.505647 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:07:03.867006 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.866979 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-c74cb6db6-m7zsv_9e541372-1023-4b32-b170-8b252dfd62f2/main/0.log" Apr 23 19:07:03.867417 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.867356 2575 generic.go:358] "Generic (PLEG): container finished" podID="9e541372-1023-4b32-b170-8b252dfd62f2" containerID="981e84c39b12d15277f47f41adc6853ff61d343622f99e611b3a2e635d5a7577" exitCode=137 Apr 23 19:07:03.867480 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.867432 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" event={"ID":"9e541372-1023-4b32-b170-8b252dfd62f2","Type":"ContainerDied","Data":"981e84c39b12d15277f47f41adc6853ff61d343622f99e611b3a2e635d5a7577"} Apr 23 19:07:03.867527 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.867475 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" event={"ID":"9e541372-1023-4b32-b170-8b252dfd62f2","Type":"ContainerDied","Data":"839bfa9568dd240f9c2542635c68a08570a4d330a29b3d9d2d8d19b092bdca3b"} Apr 23 19:07:03.867527 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.867491 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839bfa9568dd240f9c2542635c68a08570a4d330a29b3d9d2d8d19b092bdca3b" Apr 23 19:07:03.880718 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.880696 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-c74cb6db6-m7zsv_9e541372-1023-4b32-b170-8b252dfd62f2/main/0.log" Apr 23 19:07:03.881108 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:03.881084 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:07:04.026559 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.026518 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zjjj\" (UniqueName: \"kubernetes.io/projected/9e541372-1023-4b32-b170-8b252dfd62f2-kube-api-access-4zjjj\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.026752 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.026568 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-dshm\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.026752 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.026645 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-tmp-dir\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.026853 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.026777 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-home\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.026913 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.026877 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e541372-1023-4b32-b170-8b252dfd62f2-tls-certs\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.027292 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.027267 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-kserve-provision-location\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.027445 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.027327 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-model-cache\") pod \"9e541372-1023-4b32-b170-8b252dfd62f2\" (UID: \"9e541372-1023-4b32-b170-8b252dfd62f2\") " Apr 23 19:07:04.027445 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.027427 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-home" (OuterVolumeSpecName: "home") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:04.027733 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.027708 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:04.027831 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.027823 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.027948 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.027838 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.029355 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.029314 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-dshm" (OuterVolumeSpecName: "dshm") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:04.029466 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.029405 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e541372-1023-4b32-b170-8b252dfd62f2-kube-api-access-4zjjj" (OuterVolumeSpecName: "kube-api-access-4zjjj") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "kube-api-access-4zjjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:07:04.030913 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.030872 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e541372-1023-4b32-b170-8b252dfd62f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:07:04.045682 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.045628 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:04.093769 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.093716 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e541372-1023-4b32-b170-8b252dfd62f2" (UID: "9e541372-1023-4b32-b170-8b252dfd62f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:04.129362 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.129252 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.129362 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.129290 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e541372-1023-4b32-b170-8b252dfd62f2-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.129362 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.129301 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.129362 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.129311 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjjj\" (UniqueName: \"kubernetes.io/projected/9e541372-1023-4b32-b170-8b252dfd62f2-kube-api-access-4zjjj\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.129362 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.129321 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e541372-1023-4b32-b170-8b252dfd62f2-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:04.872412 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.872376 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv" Apr 23 19:07:04.902414 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.902374 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv"] Apr 23 19:07:04.906943 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:04.906909 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c74cb6db6-m7zsv"] Apr 23 19:07:05.596648 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:05.596581 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" path="/var/lib/kubelet/pods/9e541372-1023-4b32-b170-8b252dfd62f2/volumes" Apr 23 19:07:08.390230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:08.390183 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:07:08.409265 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:08.409227 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:07:08.504522 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:08.504473 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:07:09.163158 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:09.163118 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt"] Apr 23 19:07:09.163548 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:09.163485 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" containerID="cri-o://1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247" gracePeriod=30 Apr 23 19:07:18.390948 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:18.390899 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:07:18.504427 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:18.504367 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:07:28.390720 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:28.390664 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:07:28.504571 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:28.504522 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:07:38.390647 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:38.390574 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:07:38.504165 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:38.504110 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:07:39.448110 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.448083 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-b8c47cf74-mvwrt_1a546aa0-a49f-4b34-8326-080fd2c3c743/main/0.log" Apr 23 19:07:39.448499 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.448485 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:07:39.571300 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571263 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-kserve-provision-location\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.571300 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571304 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-dshm\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.571603 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571365 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a546aa0-a49f-4b34-8326-080fd2c3c743-tls-certs\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.571603 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571385 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-tmp-dir\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.571603 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571423 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-model-cache\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.571603 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571450 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wcbf\" (UniqueName: \"kubernetes.io/projected/1a546aa0-a49f-4b34-8326-080fd2c3c743-kube-api-access-9wcbf\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.571603 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571483 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-home\") pod \"1a546aa0-a49f-4b34-8326-080fd2c3c743\" (UID: \"1a546aa0-a49f-4b34-8326-080fd2c3c743\") " Apr 23 19:07:39.572041 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.571925 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:39.572393 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.572333 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-home" (OuterVolumeSpecName: "home") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:39.573879 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.573842 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a546aa0-a49f-4b34-8326-080fd2c3c743-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:07:39.574175 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.574150 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-dshm" (OuterVolumeSpecName: "dshm") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:39.574532 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.574508 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a546aa0-a49f-4b34-8326-080fd2c3c743-kube-api-access-9wcbf" (OuterVolumeSpecName: "kube-api-access-9wcbf") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "kube-api-access-9wcbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:07:39.592467 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.592421 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:39.665435 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.665392 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a546aa0-a49f-4b34-8326-080fd2c3c743" (UID: "1a546aa0-a49f-4b34-8326-080fd2c3c743"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:07:39.672726 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672691 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:39.672726 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672729 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a546aa0-a49f-4b34-8326-080fd2c3c743-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:39.672937 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672745 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:39.672937 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672759 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:39.672937 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672774 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wcbf\" (UniqueName: \"kubernetes.io/projected/1a546aa0-a49f-4b34-8326-080fd2c3c743-kube-api-access-9wcbf\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:39.672937 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672788 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:39.672937 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:39.672802 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a546aa0-a49f-4b34-8326-080fd2c3c743-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:07:40.014672 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.014642 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-b8c47cf74-mvwrt_1a546aa0-a49f-4b34-8326-080fd2c3c743/main/0.log" Apr 23 19:07:40.015046 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.015021 2575 generic.go:358] "Generic (PLEG): container finished" podID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerID="1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247" exitCode=137 Apr 23 19:07:40.015142 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.015109 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" Apr 23 19:07:40.015209 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.015108 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" event={"ID":"1a546aa0-a49f-4b34-8326-080fd2c3c743","Type":"ContainerDied","Data":"1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247"} Apr 23 19:07:40.015268 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.015224 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt" event={"ID":"1a546aa0-a49f-4b34-8326-080fd2c3c743","Type":"ContainerDied","Data":"3ff2fa315ef313e3690b6a7fbefcf6e208e9d223258bd7520852049081b2061e"} Apr 23 19:07:40.015268 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.015240 2575 scope.go:117] "RemoveContainer" containerID="1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247" Apr 23 19:07:40.034350 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.034318 2575 scope.go:117] "RemoveContainer" containerID="06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5" Apr 23 19:07:40.043912 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.043875 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt"] Apr 23 19:07:40.048127 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.048092 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-b8c47cf74-mvwrt"] Apr 23 19:07:40.104458 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.104431 2575 scope.go:117] "RemoveContainer" containerID="1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247" Apr 23 19:07:40.104869 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:07:40.104841 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247\": container with ID starting with 1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247 not found: ID does not exist" containerID="1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247" Apr 23 19:07:40.104950 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.104882 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247"} err="failed to get container status \"1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247\": rpc error: code = NotFound desc = could not find container \"1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247\": container with ID starting with 1af79526ab5810bc35038e287d4626890663d887ad0aa40f513fb803ba24a247 not found: ID does not exist" Apr 23 19:07:40.104950 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.104901 2575 scope.go:117] "RemoveContainer" containerID="06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5" Apr 23 19:07:40.105252 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:07:40.105235 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5\": container with ID starting with 06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5 not found: ID does not exist" containerID="06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5" Apr 23 19:07:40.105300 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:40.105255 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5"} err="failed to get container status \"06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5\": rpc error: code = NotFound desc = could not find container \"06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5\": container with ID starting with 06cc91f70f6a593f99cdd3d51e607be0d6e9979f11d7aad255509c9138f707a5 not found: ID does not exist" Apr 23 19:07:41.597046 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:41.597009 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" path="/var/lib/kubelet/pods/1a546aa0-a49f-4b34-8326-080fd2c3c743/volumes" Apr 23 19:07:48.390946 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:48.390889 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:07:48.504486 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:48.504436 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:07:58.390213 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:58.390156 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:07:58.504099 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:07:58.504043 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:08:08.390604 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:08.390533 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:08:08.504265 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:08.504224 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:08:18.390939 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:18.390888 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:08:18.504038 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:18.503984 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:08:22.756211 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756176 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg"] Apr 23 19:08:22.756723 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756707 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="storage-initializer" Apr 23 19:08:22.756774 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756726 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="storage-initializer" Apr 23 19:08:22.756774 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756753 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" Apr 23 19:08:22.756774 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756763 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" Apr 23 19:08:22.756873 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756779 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="storage-initializer" Apr 23 19:08:22.756873 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756789 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="storage-initializer" Apr 23 19:08:22.756873 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756806 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" Apr 23 19:08:22.756873 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756815 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" Apr 23 19:08:22.757002 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756896 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="9e541372-1023-4b32-b170-8b252dfd62f2" containerName="main" Apr 23 19:08:22.757002 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.756907 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a546aa0-a49f-4b34-8326-080fd2c3c743" containerName="main" Apr 23 19:08:22.760612 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.760574 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.762849 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.762831 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\"" Apr 23 19:08:22.769521 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.769493 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg"] Apr 23 19:08:22.861946 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.861898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.861946 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.861950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2dj\" (UniqueName: \"kubernetes.io/projected/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kube-api-access-jx2dj\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.862199 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.861984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.862199 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.862068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.862199 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.862143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.862320 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.862259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.862320 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.862292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963493 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963436 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963721 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963560 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963721 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963602 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963721 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963652 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963721 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963678 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2dj\" (UniqueName: \"kubernetes.io/projected/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kube-api-access-jx2dj\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963721 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963703 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963993 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963736 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963993 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963902 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963993 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963954 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.963993 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.963985 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.964246 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.964216 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.966065 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.966045 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.966233 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.966212 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:22.973224 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:22.973198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2dj\" (UniqueName: \"kubernetes.io/projected/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kube-api-access-jx2dj\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:23.072045 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:23.071943 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:23.224171 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:23.224141 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg"] Apr 23 19:08:23.226238 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:08:23.226206 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7526c47a_1e88_49a7_b7b7_b6bb50e80fd8.slice/crio-5f690d3eca3de56328ed66965fcc6a7dc8c890b022b8a3a3fd22f3b19f1dc68d WatchSource:0}: Error finding container 5f690d3eca3de56328ed66965fcc6a7dc8c890b022b8a3a3fd22f3b19f1dc68d: Status 404 returned error can't find the container with id 5f690d3eca3de56328ed66965fcc6a7dc8c890b022b8a3a3fd22f3b19f1dc68d Apr 23 19:08:24.181149 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:24.181051 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" event={"ID":"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8","Type":"ContainerStarted","Data":"1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166"} Apr 23 19:08:24.181149 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:24.181100 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" event={"ID":"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8","Type":"ContainerStarted","Data":"5f690d3eca3de56328ed66965fcc6a7dc8c890b022b8a3a3fd22f3b19f1dc68d"} Apr 23 19:08:28.195828 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:28.195787 2575 generic.go:358] "Generic (PLEG): container finished" podID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerID="1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166" exitCode=0 Apr 23 19:08:28.196229 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:28.195857 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" event={"ID":"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8","Type":"ContainerDied","Data":"1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166"} Apr 23 19:08:28.390764 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:28.390713 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:08:28.504292 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:28.504254 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:08:29.202210 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:29.202167 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" event={"ID":"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8","Type":"ContainerStarted","Data":"07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab"} Apr 23 19:08:29.232743 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:29.232689 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podStartSLOduration=7.232673918 podStartE2EDuration="7.232673918s" podCreationTimestamp="2026-04-23 19:08:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:08:29.231062646 +0000 UTC m=+5208.250515660" watchObservedRunningTime="2026-04-23 19:08:29.232673918 +0000 UTC m=+5208.252126933" Apr 23 19:08:33.072764 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:33.072711 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:33.073167 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:33.072812 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:08:33.074431 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:33.074405 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:08:38.390743 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:38.390692 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:08:38.504712 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:38.504656 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:08:43.072937 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:43.072885 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:08:48.390115 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:48.390066 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:08:48.504335 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:48.504282 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:08:53.072701 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:53.072660 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:08:58.391044 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:58.390992 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:08:58.504353 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:08:58.504300 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:09:03.072743 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:03.072688 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:09:08.390193 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:08.390138 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused" Apr 23 19:09:08.503756 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:08.503690 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused" Apr 23 19:09:13.073232 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:13.073174 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:09:18.405291 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:18.405254 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:09:18.417312 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:18.417281 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:09:18.513675 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:18.513638 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:09:18.523007 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:18.522974 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:09:23.073271 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:23.073208 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:09:33.072362 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:33.072318 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:09:33.222925 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:33.222860 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9"] Apr 23 19:09:33.224019 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:33.223979 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" containerID="cri-o://c013baf683166c86306d551670d0cddd8208ed261cf00d241180f5d301bbf867" gracePeriod=30 Apr 23 19:09:33.225926 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:33.225893 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr"] Apr 23 19:09:33.226294 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:33.226246 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" containerID="cri-o://fd04574716781999cc2cae77415a7bd36fe2015b7106a3006cee9f787512bdf4" gracePeriod=30 Apr 23 19:09:43.072559 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:43.072500 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:09:52.311534 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.311478 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg"] Apr 23 19:09:52.315410 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.315382 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.318237 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.318208 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-zspl4\"" Apr 23 19:09:52.318557 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.318532 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" Apr 23 19:09:52.334820 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.334742 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg"] Apr 23 19:09:52.337368 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.337331 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545"] Apr 23 19:09:52.341896 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.341872 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.349046 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.349010 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545"] Apr 23 19:09:52.408846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.408787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8739b531-9c81-469e-8f3c-3b6682d00937-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.408846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.408843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-home\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.409125 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.408912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698575f0-3e94-40e2-962f-8398eac7549a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.409125 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.408978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.409125 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.409125 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77n4\" (UniqueName: \"kubernetes.io/projected/698575f0-3e94-40e2-962f-8398eac7549a-kube-api-access-t77n4\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.409125 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.409125 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.409461 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-dshm\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.409461 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.409461 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2rs\" (UniqueName: \"kubernetes.io/projected/8739b531-9c81-469e-8f3c-3b6682d00937-kube-api-access-dz2rs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.409461 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.409461 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.409461 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.409277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510148 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510108 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510148 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510144 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510177 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8739b531-9c81-469e-8f3c-3b6682d00937-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510200 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-home\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510216 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698575f0-3e94-40e2-962f-8398eac7549a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510256 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510280 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510310 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t77n4\" (UniqueName: \"kubernetes.io/projected/698575f0-3e94-40e2-962f-8398eac7549a-kube-api-access-t77n4\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510344 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510360 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510425 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510418 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-dshm\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510461 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510492 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2rs\" (UniqueName: \"kubernetes.io/projected/8739b531-9c81-469e-8f3c-3b6682d00937-kube-api-access-dz2rs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510524 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510540 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510670 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.510944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.510815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.511316 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.511018 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.511316 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.511100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.511316 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.511159 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.511316 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.511229 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-home\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.513198 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.513172 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.513483 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.513460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698575f0-3e94-40e2-962f-8398eac7549a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.513612 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.513557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-dshm\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.513870 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.513842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8739b531-9c81-469e-8f3c-3b6682d00937-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.519667 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.519636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2rs\" (UniqueName: \"kubernetes.io/projected/8739b531-9c81-469e-8f3c-3b6682d00937-kube-api-access-dz2rs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.519796 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.519644 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77n4\" (UniqueName: \"kubernetes.io/projected/698575f0-3e94-40e2-962f-8398eac7549a-kube-api-access-t77n4\") pod \"custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.626877 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.626772 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:52.655287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.655227 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:09:52.795663 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.795617 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg"] Apr 23 19:09:52.797827 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:09:52.797797 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698575f0_3e94_40e2_962f_8398eac7549a.slice/crio-d23646ea1573dc15f28c16aa45e0aa45648d9b7428a99f12e7b174e3201fee5f WatchSource:0}: Error finding container d23646ea1573dc15f28c16aa45e0aa45648d9b7428a99f12e7b174e3201fee5f: Status 404 returned error can't find the container with id d23646ea1573dc15f28c16aa45e0aa45648d9b7428a99f12e7b174e3201fee5f Apr 23 19:09:52.799788 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.799764 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 19:09:52.816794 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:52.816759 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545"] Apr 23 19:09:52.817327 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:09:52.817302 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8739b531_9c81_469e_8f3c_3b6682d00937.slice/crio-230d64d9cc0c28e303a69594f44cc3acec328bcda3de51baa18ee2e4f447a80a WatchSource:0}: Error finding container 230d64d9cc0c28e303a69594f44cc3acec328bcda3de51baa18ee2e4f447a80a: Status 404 returned error can't find the container with id 230d64d9cc0c28e303a69594f44cc3acec328bcda3de51baa18ee2e4f447a80a Apr 23 19:09:53.073043 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:53.072992 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:09:53.520653 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:53.520567 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerStarted","Data":"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367"} Apr 23 19:09:53.520653 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:53.520638 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerStarted","Data":"d23646ea1573dc15f28c16aa45e0aa45648d9b7428a99f12e7b174e3201fee5f"} Apr 23 19:09:53.521206 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:53.520754 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:09:53.522604 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:53.522556 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" event={"ID":"8739b531-9c81-469e-8f3c-3b6682d00937","Type":"ContainerStarted","Data":"186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a"} Apr 23 19:09:53.522708 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:53.522619 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" event={"ID":"8739b531-9c81-469e-8f3c-3b6682d00937","Type":"ContainerStarted","Data":"230d64d9cc0c28e303a69594f44cc3acec328bcda3de51baa18ee2e4f447a80a"} Apr 23 19:09:54.529781 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:54.529649 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerStarted","Data":"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10"} Apr 23 19:09:58.547701 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:58.547659 2575 generic.go:358] "Generic (PLEG): container finished" podID="698575f0-3e94-40e2-962f-8398eac7549a" containerID="1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10" exitCode=0 Apr 23 19:09:58.548106 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:58.547729 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerDied","Data":"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10"} Apr 23 19:09:59.554355 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:59.554317 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerStarted","Data":"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1"} Apr 23 19:09:59.586312 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:09:59.586256 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podStartSLOduration=7.586238145 podStartE2EDuration="7.586238145s" podCreationTimestamp="2026-04-23 19:09:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:09:59.58040168 +0000 UTC m=+5298.599854696" watchObservedRunningTime="2026-04-23 19:09:59.586238145 +0000 UTC m=+5298.605691161" Apr 23 19:10:02.627354 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:02.627302 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:10:02.627769 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:02.627370 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:10:02.629098 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:02.629056 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:10:02.647262 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:02.647229 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:10:03.073325 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.073273 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:10:03.224114 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.224027 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="llm-d-routing-sidecar" containerID="cri-o://4f752c18d541fce3d8709cdf5125823a4b04bd997c313ad32a871ee331735382" gracePeriod=2 Apr 23 19:10:03.572494 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.572456 2575 generic.go:358] "Generic (PLEG): container finished" podID="9259e06d-020f-4a51-a5e4-f116d363a173" containerID="fd04574716781999cc2cae77415a7bd36fe2015b7106a3006cee9f787512bdf4" exitCode=137 Apr 23 19:10:03.572711 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.572541 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" event={"ID":"9259e06d-020f-4a51-a5e4-f116d363a173","Type":"ContainerDied","Data":"fd04574716781999cc2cae77415a7bd36fe2015b7106a3006cee9f787512bdf4"} Apr 23 19:10:03.574456 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.574434 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9_367a7f17-7395-4728-b2ff-486dcae24265/main/0.log" Apr 23 19:10:03.575131 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.575102 2575 generic.go:358] "Generic (PLEG): container finished" podID="367a7f17-7395-4728-b2ff-486dcae24265" containerID="c013baf683166c86306d551670d0cddd8208ed261cf00d241180f5d301bbf867" exitCode=137 Apr 23 19:10:03.575131 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.575128 2575 generic.go:358] "Generic (PLEG): container finished" podID="367a7f17-7395-4728-b2ff-486dcae24265" containerID="4f752c18d541fce3d8709cdf5125823a4b04bd997c313ad32a871ee331735382" exitCode=0 Apr 23 19:10:03.575327 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.575174 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerDied","Data":"c013baf683166c86306d551670d0cddd8208ed261cf00d241180f5d301bbf867"} Apr 23 19:10:03.575327 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.575231 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerDied","Data":"4f752c18d541fce3d8709cdf5125823a4b04bd997c313ad32a871ee331735382"} Apr 23 19:10:03.615289 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.615257 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9_367a7f17-7395-4728-b2ff-486dcae24265/main/0.log" Apr 23 19:10:03.616110 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.616088 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:10:03.619623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.619581 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:10:03.728494 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728459 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-dshm\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728518 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-kserve-provision-location\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728548 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367a7f17-7395-4728-b2ff-486dcae24265-tls-certs\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728580 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-model-cache\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728641 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw8vz\" (UniqueName: \"kubernetes.io/projected/367a7f17-7395-4728-b2ff-486dcae24265-kube-api-access-pw8vz\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728671 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-kserve-provision-location\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728704 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-tmp-dir\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728738 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-tmp-dir\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728760 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-model-cache\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.729031 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.728824 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-dshm\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729504 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.729039 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bfjp\" (UniqueName: \"kubernetes.io/projected/9259e06d-020f-4a51-a5e4-f116d363a173-kube-api-access-6bfjp\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.729504 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.729107 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9259e06d-020f-4a51-a5e4-f116d363a173-tls-certs\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.729504 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.729117 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-model-cache" (OuterVolumeSpecName: "model-cache") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.729504 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.729188 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-home\") pod \"367a7f17-7395-4728-b2ff-486dcae24265\" (UID: \"367a7f17-7395-4728-b2ff-486dcae24265\") " Apr 23 19:10:03.729504 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.729247 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-home\") pod \"9259e06d-020f-4a51-a5e4-f116d363a173\" (UID: \"9259e06d-020f-4a51-a5e4-f116d363a173\") " Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.731627 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-model-cache" (OuterVolumeSpecName: "model-cache") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.734160 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9259e06d-020f-4a51-a5e4-f116d363a173-kube-api-access-6bfjp" (OuterVolumeSpecName: "kube-api-access-6bfjp") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "kube-api-access-6bfjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.734575 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367a7f17-7395-4728-b2ff-486dcae24265-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.735191 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-home" (OuterVolumeSpecName: "home") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.738397 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-dshm" (OuterVolumeSpecName: "dshm") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.738501 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367a7f17-7395-4728-b2ff-486dcae24265-kube-api-access-pw8vz" (OuterVolumeSpecName: "kube-api-access-pw8vz") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "kube-api-access-pw8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.738651 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9259e06d-020f-4a51-a5e4-f116d363a173-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.738963 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367a7f17-7395-4728-b2ff-486dcae24265-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.738993 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739008 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8vz\" (UniqueName: \"kubernetes.io/projected/367a7f17-7395-4728-b2ff-486dcae24265-kube-api-access-pw8vz\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739021 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739036 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739054 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bfjp\" (UniqueName: \"kubernetes.io/projected/9259e06d-020f-4a51-a5e4-f116d363a173-kube-api-access-6bfjp\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739068 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9259e06d-020f-4a51-a5e4-f116d363a173-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739083 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739705 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-dshm" (OuterVolumeSpecName: "dshm") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.739817 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-home" (OuterVolumeSpecName: "home") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.747547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.746953 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.755201 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.755099 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.799519 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.799460 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9259e06d-020f-4a51-a5e4-f116d363a173" (UID: "9259e06d-020f-4a51-a5e4-f116d363a173"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.808238 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.808186 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "367a7f17-7395-4728-b2ff-486dcae24265" (UID: "367a7f17-7395-4728-b2ff-486dcae24265"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:03.840419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.840380 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.840419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.840413 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.840419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.840424 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.840419 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.840433 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.841060 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.840441 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9259e06d-020f-4a51-a5e4-f116d363a173-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:03.841060 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:03.840450 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/367a7f17-7395-4728-b2ff-486dcae24265-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:04.580978 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.580834 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" event={"ID":"9259e06d-020f-4a51-a5e4-f116d363a173","Type":"ContainerDied","Data":"6da5382f6e95eb6c303738dad5bfc15b1cd7ed012e133f2102f28601857c1c72"} Apr 23 19:10:04.580978 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.580864 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr" Apr 23 19:10:04.580978 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.580902 2575 scope.go:117] "RemoveContainer" containerID="fd04574716781999cc2cae77415a7bd36fe2015b7106a3006cee9f787512bdf4" Apr 23 19:10:04.582780 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.582749 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9_367a7f17-7395-4728-b2ff-486dcae24265/main/0.log" Apr 23 19:10:04.583574 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.583545 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" event={"ID":"367a7f17-7395-4728-b2ff-486dcae24265","Type":"ContainerDied","Data":"5c0d400074f3962fb706251cd95a08eb905ce904bd32c8f28b19007ff7a06e25"} Apr 23 19:10:04.583748 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.583729 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9" Apr 23 19:10:04.592924 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.592903 2575 scope.go:117] "RemoveContainer" containerID="481040fb03b31edb59876c30979e8e7e0ec11eee24c4202cc6ccbd1b2441de0f" Apr 23 19:10:04.606163 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.606132 2575 scope.go:117] "RemoveContainer" containerID="c013baf683166c86306d551670d0cddd8208ed261cf00d241180f5d301bbf867" Apr 23 19:10:04.607861 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.607816 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr"] Apr 23 19:10:04.613649 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.613614 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-84pljnr"] Apr 23 19:10:04.617939 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.617910 2575 scope.go:117] "RemoveContainer" containerID="d64a193588831842be654fe80110505a0f854f7ae45ea376c0b60e989ce8504c" Apr 23 19:10:04.628863 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.628823 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9"] Apr 23 19:10:04.632454 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.632429 2575 scope.go:117] "RemoveContainer" containerID="4f752c18d541fce3d8709cdf5125823a4b04bd997c313ad32a871ee331735382" Apr 23 19:10:04.635376 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:04.635345 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f84cb79f7-tppf9"] Apr 23 19:10:05.595938 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:05.595906 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="367a7f17-7395-4728-b2ff-486dcae24265" path="/var/lib/kubelet/pods/367a7f17-7395-4728-b2ff-486dcae24265/volumes" Apr 23 19:10:05.596405 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:05.596390 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" path="/var/lib/kubelet/pods/9259e06d-020f-4a51-a5e4-f116d363a173/volumes" Apr 23 19:10:12.627404 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:12.627349 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:10:13.073077 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:13.073021 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused" Apr 23 19:10:22.628128 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:22.628074 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:10:23.089015 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:23.088976 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:10:23.097919 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:23.097883 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:10:28.679306 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:28.679267 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg"] Apr 23 19:10:28.679850 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:28.679531 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" containerID="cri-o://07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab" gracePeriod=30 Apr 23 19:10:32.628158 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:32.628096 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:10:35.273021 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.272981 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273418 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273443 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="storage-initializer" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273449 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="storage-initializer" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273469 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273477 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="storage-initializer" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273482 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="storage-initializer" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273489 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="llm-d-routing-sidecar" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273494 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="llm-d-routing-sidecar" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273559 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="9259e06d-020f-4a51-a5e4-f116d363a173" containerName="main" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273573 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="llm-d-routing-sidecar" Apr 23 19:10:35.273763 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.273583 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="367a7f17-7395-4728-b2ff-486dcae24265" containerName="main" Apr 23 19:10:35.277209 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.277180 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.279698 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.279672 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-kgc5v\"" Apr 23 19:10:35.279825 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.279675 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\"" Apr 23 19:10:35.289519 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.289490 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 23 19:10:35.429084 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvtt7\" (UniqueName: \"kubernetes.io/projected/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kube-api-access-bvtt7\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.429279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.429279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.429279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.429279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.429279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.429279 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.429265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530194 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530098 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530194 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530155 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530194 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530178 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530470 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530209 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bvtt7\" (UniqueName: \"kubernetes.io/projected/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kube-api-access-bvtt7\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530470 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530255 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530470 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530307 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530470 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530336 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530705 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530705 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530705 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.530828 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.530798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.532568 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.532547 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.532738 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.532718 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.542546 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.542518 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvtt7\" (UniqueName: \"kubernetes.io/projected/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kube-api-access-bvtt7\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.590643 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.590596 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:35.740710 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:35.740484 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 23 19:10:35.743504 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:10:35.743476 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ea5617_3af1_43f2_8d98_2aeb8e37526d.slice/crio-2ff5e46b424c1837e73e8f25ca9cb56d31a59467e641c96e9d306f5d5430aef6 WatchSource:0}: Error finding container 2ff5e46b424c1837e73e8f25ca9cb56d31a59467e641c96e9d306f5d5430aef6: Status 404 returned error can't find the container with id 2ff5e46b424c1837e73e8f25ca9cb56d31a59467e641c96e9d306f5d5430aef6 Apr 23 19:10:36.718204 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:36.718138 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b8ea5617-3af1-43f2-8d98-2aeb8e37526d","Type":"ContainerStarted","Data":"6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5"} Apr 23 19:10:36.718204 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:36.718187 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b8ea5617-3af1-43f2-8d98-2aeb8e37526d","Type":"ContainerStarted","Data":"2ff5e46b424c1837e73e8f25ca9cb56d31a59467e641c96e9d306f5d5430aef6"} Apr 23 19:10:40.734494 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:40.734459 2575 generic.go:358] "Generic (PLEG): container finished" podID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerID="6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5" exitCode=0 Apr 23 19:10:40.734901 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:40.734531 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b8ea5617-3af1-43f2-8d98-2aeb8e37526d","Type":"ContainerDied","Data":"6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5"} Apr 23 19:10:41.740278 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:41.740240 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b8ea5617-3af1-43f2-8d98-2aeb8e37526d","Type":"ContainerStarted","Data":"d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6"} Apr 23 19:10:41.764713 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:41.764642 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=6.76462234 podStartE2EDuration="6.76462234s" podCreationTimestamp="2026-04-23 19:10:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:10:41.761972188 +0000 UTC m=+5340.781425205" watchObservedRunningTime="2026-04-23 19:10:41.76462234 +0000 UTC m=+5340.784075357" Apr 23 19:10:41.895501 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:41.895462 2575 scope.go:117] "RemoveContainer" containerID="7c996a7f78d1e3b7c51482e9dd1a1e016ce91b1cf6f9365b66169059d7486136" Apr 23 19:10:41.961879 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:41.961845 2575 scope.go:117] "RemoveContainer" containerID="981e84c39b12d15277f47f41adc6853ff61d343622f99e611b3a2e635d5a7577" Apr 23 19:10:42.628306 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:42.628258 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:10:45.592622 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:45.592558 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:10:45.596343 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:45.596314 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:10:52.627944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:52.627851 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:10:55.591466 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:55.591395 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:10:59.001175 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.001141 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg_7526c47a-1e88-49a7-b7b7-b6bb50e80fd8/main/0.log" Apr 23 19:10:59.001640 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.001577 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:10:59.052349 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052316 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tmp-dir\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052349 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052353 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-home\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052652 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052385 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-model-cache\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052652 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052430 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kserve-provision-location\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052652 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052458 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx2dj\" (UniqueName: \"kubernetes.io/projected/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kube-api-access-jx2dj\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052652 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052508 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-dshm\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052652 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052567 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tls-certs\") pod \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\" (UID: \"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8\") " Apr 23 19:10:59.052935 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052691 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:59.052935 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.052867 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.055323 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.053033 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-home" (OuterVolumeSpecName: "home") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:59.056124 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.055970 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kube-api-access-jx2dj" (OuterVolumeSpecName: "kube-api-access-jx2dj") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "kube-api-access-jx2dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:10:59.056486 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.056449 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:10:59.056776 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.056735 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-dshm" (OuterVolumeSpecName: "dshm") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:59.070438 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.070393 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:59.114282 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.114233 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" (UID: "7526c47a-1e88-49a7-b7b7-b6bb50e80fd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:10:59.154275 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.154233 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.154275 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.154270 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2dj\" (UniqueName: \"kubernetes.io/projected/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-kube-api-access-jx2dj\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.154275 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.154284 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.154524 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.154297 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.154524 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.154308 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.154524 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.154319 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:10:59.816262 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.816225 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg_7526c47a-1e88-49a7-b7b7-b6bb50e80fd8/main/0.log" Apr 23 19:10:59.816677 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.816641 2575 generic.go:358] "Generic (PLEG): container finished" podID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerID="07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab" exitCode=137 Apr 23 19:10:59.816802 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.816724 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" event={"ID":"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8","Type":"ContainerDied","Data":"07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab"} Apr 23 19:10:59.816802 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.816771 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" event={"ID":"7526c47a-1e88-49a7-b7b7-b6bb50e80fd8","Type":"ContainerDied","Data":"5f690d3eca3de56328ed66965fcc6a7dc8c890b022b8a3a3fd22f3b19f1dc68d"} Apr 23 19:10:59.816802 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.816739 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg" Apr 23 19:10:59.816802 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.816792 2575 scope.go:117] "RemoveContainer" containerID="07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab" Apr 23 19:10:59.827359 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.827333 2575 scope.go:117] "RemoveContainer" containerID="1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166" Apr 23 19:10:59.842867 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.842832 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg"] Apr 23 19:10:59.846862 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.846832 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6d59c9877dfckvg"] Apr 23 19:10:59.893185 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.893155 2575 scope.go:117] "RemoveContainer" containerID="07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab" Apr 23 19:10:59.893520 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:10:59.893495 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab\": container with ID starting with 07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab not found: ID does not exist" containerID="07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab" Apr 23 19:10:59.893650 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.893531 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab"} err="failed to get container status \"07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab\": rpc error: code = NotFound desc = could not find container \"07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab\": container with ID starting with 07b4938106d64a159c7e824773a22bcee135161705eb4a2745c832b0dd62f0ab not found: ID does not exist" Apr 23 19:10:59.893650 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.893554 2575 scope.go:117] "RemoveContainer" containerID="1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166" Apr 23 19:10:59.893909 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:10:59.893892 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166\": container with ID starting with 1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166 not found: ID does not exist" containerID="1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166" Apr 23 19:10:59.893975 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:10:59.893921 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166"} err="failed to get container status \"1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166\": rpc error: code = NotFound desc = could not find container \"1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166\": container with ID starting with 1a33f05d5a1aba3ec8db3e9995028fa8e936c2d8d4e203fe6eb938776cbb3166 not found: ID does not exist" Apr 23 19:11:01.597073 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:01.597024 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" path="/var/lib/kubelet/pods/7526c47a-1e88-49a7-b7b7-b6bb50e80fd8/volumes" Apr 23 19:11:02.627301 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:02.627248 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:11:05.591144 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:05.591084 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:11:05.596526 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:05.596489 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:11:12.628064 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:12.628009 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:11:15.592022 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:15.591975 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:11:22.628231 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:22.628170 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:11:24.923896 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:24.923842 2575 generic.go:358] "Generic (PLEG): container finished" podID="8739b531-9c81-469e-8f3c-3b6682d00937" containerID="186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a" exitCode=0 Apr 23 19:11:24.924281 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:24.923927 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" event={"ID":"8739b531-9c81-469e-8f3c-3b6682d00937","Type":"ContainerDied","Data":"186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a"} Apr 23 19:11:25.591087 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:25.591039 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:11:25.929655 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:25.929540 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" event={"ID":"8739b531-9c81-469e-8f3c-3b6682d00937","Type":"ContainerStarted","Data":"a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209"} Apr 23 19:11:25.958245 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:25.958179 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podStartSLOduration=93.958158422 podStartE2EDuration="1m33.958158422s" podCreationTimestamp="2026-04-23 19:09:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:11:25.954580506 +0000 UTC m=+5384.974033521" watchObservedRunningTime="2026-04-23 19:11:25.958158422 +0000 UTC m=+5384.977611438" Apr 23 19:11:32.628267 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:32.628214 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused" Apr 23 19:11:32.656305 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:32.656264 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:11:32.656518 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:32.656319 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:11:32.658310 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:32.658274 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:11:35.591369 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:35.591318 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:11:41.835412 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:41.835284 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:11:41.845545 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:41.845520 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:11:42.638401 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:42.638363 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:11:42.654252 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:42.654221 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:11:42.656517 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:42.656485 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:11:45.591290 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:45.591245 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:11:52.655801 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:52.655759 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:11:55.592094 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:11:55.592033 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:12:02.656320 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:02.656259 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:12:05.591197 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:05.591144 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:12:12.656045 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:12.656002 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:12:15.591946 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:15.591890 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Apr 23 19:12:22.655881 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:22.655839 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:12:25.602168 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:25.602136 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:12:25.610150 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:25.610114 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:12:32.655952 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:32.655897 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:12:37.803999 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:37.803956 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 23 19:12:37.804514 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:37.804350 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" containerID="cri-o://d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6" gracePeriod=30 Apr 23 19:12:38.877757 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:38.877725 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:12:39.018085 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018040 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvtt7\" (UniqueName: \"kubernetes.io/projected/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kube-api-access-bvtt7\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018111 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kserve-provision-location\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018138 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-model-cache\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018165 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tmp-dir\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018197 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-dshm\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018220 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tls-certs\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018574 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018333 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-home\") pod \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\" (UID: \"b8ea5617-3af1-43f2-8d98-2aeb8e37526d\") " Apr 23 19:12:39.018574 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018450 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:12:39.018713 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.018636 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.019105 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.019066 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-home" (OuterVolumeSpecName: "home") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:12:39.021005 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.020970 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:12:39.021005 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.020970 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-dshm" (OuterVolumeSpecName: "dshm") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:12:39.021791 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.021764 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kube-api-access-bvtt7" (OuterVolumeSpecName: "kube-api-access-bvtt7") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "kube-api-access-bvtt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:12:39.030177 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.030112 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:12:39.090098 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.090036 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8ea5617-3af1-43f2-8d98-2aeb8e37526d" (UID: "b8ea5617-3af1-43f2-8d98-2aeb8e37526d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:12:39.120002 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.119967 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvtt7\" (UniqueName: \"kubernetes.io/projected/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kube-api-access-bvtt7\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.120002 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.120000 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.120216 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.120013 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.120216 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.120028 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.120216 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.120040 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.120216 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.120051 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea5617-3af1-43f2-8d98-2aeb8e37526d-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:12:39.227026 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.226992 2575 generic.go:358] "Generic (PLEG): container finished" podID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerID="d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6" exitCode=0 Apr 23 19:12:39.227206 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.227039 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b8ea5617-3af1-43f2-8d98-2aeb8e37526d","Type":"ContainerDied","Data":"d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6"} Apr 23 19:12:39.227206 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.227063 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b8ea5617-3af1-43f2-8d98-2aeb8e37526d","Type":"ContainerDied","Data":"2ff5e46b424c1837e73e8f25ca9cb56d31a59467e641c96e9d306f5d5430aef6"} Apr 23 19:12:39.227206 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.227079 2575 scope.go:117] "RemoveContainer" containerID="d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6" Apr 23 19:12:39.227206 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.227093 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 23 19:12:39.235897 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.235873 2575 scope.go:117] "RemoveContainer" containerID="6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5" Apr 23 19:12:39.250471 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.250440 2575 scope.go:117] "RemoveContainer" containerID="d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6" Apr 23 19:12:39.250820 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:12:39.250788 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6\": container with ID starting with d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6 not found: ID does not exist" containerID="d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6" Apr 23 19:12:39.250904 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.250835 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6"} err="failed to get container status \"d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6\": rpc error: code = NotFound desc = could not find container \"d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6\": container with ID starting with d126eb34cfd16ce01d1f62fc852a3a3e00734b266c52fea19800fc4458943aa6 not found: ID does not exist" Apr 23 19:12:39.250904 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.250863 2575 scope.go:117] "RemoveContainer" containerID="6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5" Apr 23 19:12:39.251184 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:12:39.251161 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5\": container with ID starting with 6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5 not found: ID does not exist" containerID="6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5" Apr 23 19:12:39.251236 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.251189 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5"} err="failed to get container status \"6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5\": rpc error: code = NotFound desc = could not find container \"6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5\": container with ID starting with 6b3a78d5a1fe78b85933638be2c414a7979e43cf4f56af9c25e87274116802f5 not found: ID does not exist" Apr 23 19:12:39.253948 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.253919 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 23 19:12:39.259010 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.258979 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 23 19:12:39.595943 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:39.595903 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" path="/var/lib/kubelet/pods/b8ea5617-3af1-43f2-8d98-2aeb8e37526d/volumes" Apr 23 19:12:42.656395 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:42.656348 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:12:52.656252 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:12:52.656210 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:13:02.655636 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:02.655510 2575 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused" Apr 23 19:13:12.665547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:12.665509 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:13:12.673441 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:12.673411 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:13:24.512610 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:24.512556 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545"] Apr 23 19:13:24.513242 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:24.513078 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" containerID="cri-o://a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209" gracePeriod=30 Apr 23 19:13:24.518434 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:24.518407 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg"] Apr 23 19:13:24.518784 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:24.518754 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" containerID="cri-o://0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1" gracePeriod=30 Apr 23 19:13:54.519061 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:54.519019 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="llm-d-routing-sidecar" containerID="cri-o://7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367" gracePeriod=2 Apr 23 19:13:54.924127 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:54.924093 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:13:54.948848 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:54.948816 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg_698575f0-3e94-40e2-962f-8398eac7549a/main/0.log" Apr 23 19:13:54.949498 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:54.949479 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:13:55.017990 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.017950 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz2rs\" (UniqueName: \"kubernetes.io/projected/8739b531-9c81-469e-8f3c-3b6682d00937-kube-api-access-dz2rs\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018196 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018012 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-home\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018196 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018042 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-home\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018196 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018076 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-tmp-dir\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018196 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018124 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-dshm\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018196 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018165 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-model-cache\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018200 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-dshm\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018248 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8739b531-9c81-469e-8f3c-3b6682d00937-tls-certs\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018273 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-kserve-provision-location\") pod \"8739b531-9c81-469e-8f3c-3b6682d00937\" (UID: \"8739b531-9c81-469e-8f3c-3b6682d00937\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018296 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t77n4\" (UniqueName: \"kubernetes.io/projected/698575f0-3e94-40e2-962f-8398eac7549a-kube-api-access-t77n4\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018321 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-tmp-dir\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018346 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-kserve-provision-location\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018389 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698575f0-3e94-40e2-962f-8398eac7549a-tls-certs\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018538 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018413 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-model-cache\") pod \"698575f0-3e94-40e2-962f-8398eac7549a\" (UID: \"698575f0-3e94-40e2-962f-8398eac7549a\") " Apr 23 19:13:55.018968 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.018842 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-home" (OuterVolumeSpecName: "home") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.019052 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.019015 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-model-cache" (OuterVolumeSpecName: "model-cache") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.019114 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.019088 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-home" (OuterVolumeSpecName: "home") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.021282 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.021255 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8739b531-9c81-469e-8f3c-3b6682d00937-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:13:55.021492 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.021410 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-model-cache" (OuterVolumeSpecName: "model-cache") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.023298 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.023263 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698575f0-3e94-40e2-962f-8398eac7549a-kube-api-access-t77n4" (OuterVolumeSpecName: "kube-api-access-t77n4") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "kube-api-access-t77n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:13:55.023454 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.023430 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-dshm" (OuterVolumeSpecName: "dshm") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.025087 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.025065 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698575f0-3e94-40e2-962f-8398eac7549a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 23 19:13:55.035109 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.035014 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8739b531-9c81-469e-8f3c-3b6682d00937-kube-api-access-dz2rs" (OuterVolumeSpecName: "kube-api-access-dz2rs") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "kube-api-access-dz2rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:13:55.036378 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.036350 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-dshm" (OuterVolumeSpecName: "dshm") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.042633 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.042601 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.042981 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.042963 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.096344 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.096298 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "698575f0-3e94-40e2-962f-8398eac7549a" (UID: "698575f0-3e94-40e2-962f-8398eac7549a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.100749 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.100713 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8739b531-9c81-469e-8f3c-3b6682d00937" (UID: "8739b531-9c81-469e-8f3c-3b6682d00937"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:13:55.120006 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.119969 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120006 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120002 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120006 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120012 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8739b531-9c81-469e-8f3c-3b6682d00937-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120021 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120032 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77n4\" (UniqueName: \"kubernetes.io/projected/698575f0-3e94-40e2-962f-8398eac7549a-kube-api-access-t77n4\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120041 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120050 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-kserve-provision-location\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120059 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698575f0-3e94-40e2-962f-8398eac7549a-tls-certs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120068 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-model-cache\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120076 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2rs\" (UniqueName: \"kubernetes.io/projected/8739b531-9c81-469e-8f3c-3b6682d00937-kube-api-access-dz2rs\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120084 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120092 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698575f0-3e94-40e2-962f-8398eac7549a-home\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120099 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-tmp-dir\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.120230 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.120107 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8739b531-9c81-469e-8f3c-3b6682d00937-dshm\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:13:55.508987 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.508958 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg_698575f0-3e94-40e2-962f-8398eac7549a/main/0.log" Apr 23 19:13:55.509547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509522 2575 generic.go:358] "Generic (PLEG): container finished" podID="698575f0-3e94-40e2-962f-8398eac7549a" containerID="0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1" exitCode=137 Apr 23 19:13:55.509547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509545 2575 generic.go:358] "Generic (PLEG): container finished" podID="698575f0-3e94-40e2-962f-8398eac7549a" containerID="7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367" exitCode=0 Apr 23 19:13:55.509699 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509618 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerDied","Data":"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1"} Apr 23 19:13:55.509699 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509658 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerDied","Data":"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367"} Apr 23 19:13:55.509699 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509674 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" event={"ID":"698575f0-3e94-40e2-962f-8398eac7549a","Type":"ContainerDied","Data":"d23646ea1573dc15f28c16aa45e0aa45648d9b7428a99f12e7b174e3201fee5f"} Apr 23 19:13:55.509699 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509675 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg" Apr 23 19:13:55.509699 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.509694 2575 scope.go:117] "RemoveContainer" containerID="0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1" Apr 23 19:13:55.511184 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.511158 2575 generic.go:358] "Generic (PLEG): container finished" podID="8739b531-9c81-469e-8f3c-3b6682d00937" containerID="a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209" exitCode=137 Apr 23 19:13:55.511291 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.511223 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" event={"ID":"8739b531-9c81-469e-8f3c-3b6682d00937","Type":"ContainerDied","Data":"a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209"} Apr 23 19:13:55.511291 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.511257 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" event={"ID":"8739b531-9c81-469e-8f3c-3b6682d00937","Type":"ContainerDied","Data":"230d64d9cc0c28e303a69594f44cc3acec328bcda3de51baa18ee2e4f447a80a"} Apr 23 19:13:55.511291 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.511262 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545" Apr 23 19:13:55.518467 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.518279 2575 scope.go:117] "RemoveContainer" containerID="1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10" Apr 23 19:13:55.538172 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.538002 2575 scope.go:117] "RemoveContainer" containerID="7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367" Apr 23 19:13:55.540544 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.540511 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545"] Apr 23 19:13:55.545169 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.545147 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5cd8c8d959-p7545"] Apr 23 19:13:55.547829 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.547810 2575 scope.go:117] "RemoveContainer" containerID="0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1" Apr 23 19:13:55.548097 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:13:55.548077 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1\": container with ID starting with 0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1 not found: ID does not exist" containerID="0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1" Apr 23 19:13:55.548178 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548111 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1"} err="failed to get container status \"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1\": rpc error: code = NotFound desc = could not find container \"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1\": container with ID starting with 0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1 not found: ID does not exist" Apr 23 19:13:55.548178 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548136 2575 scope.go:117] "RemoveContainer" containerID="1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10" Apr 23 19:13:55.548380 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:13:55.548364 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10\": container with ID starting with 1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10 not found: ID does not exist" containerID="1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10" Apr 23 19:13:55.548441 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548388 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10"} err="failed to get container status \"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10\": rpc error: code = NotFound desc = could not find container \"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10\": container with ID starting with 1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10 not found: ID does not exist" Apr 23 19:13:55.548441 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548409 2575 scope.go:117] "RemoveContainer" containerID="7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367" Apr 23 19:13:55.548694 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:13:55.548678 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367\": container with ID starting with 7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367 not found: ID does not exist" containerID="7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367" Apr 23 19:13:55.548755 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548698 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367"} err="failed to get container status \"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367\": rpc error: code = NotFound desc = could not find container \"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367\": container with ID starting with 7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367 not found: ID does not exist" Apr 23 19:13:55.548755 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548715 2575 scope.go:117] "RemoveContainer" containerID="0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1" Apr 23 19:13:55.548991 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548967 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1"} err="failed to get container status \"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1\": rpc error: code = NotFound desc = could not find container \"0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1\": container with ID starting with 0a3584984940392c35c321df9c3a71e9f69cf6fda02dd547106905f51d87ddb1 not found: ID does not exist" Apr 23 19:13:55.549043 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.548994 2575 scope.go:117] "RemoveContainer" containerID="1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10" Apr 23 19:13:55.549215 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.549194 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10"} err="failed to get container status \"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10\": rpc error: code = NotFound desc = could not find container \"1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10\": container with ID starting with 1d52c6a9c5901b5c3682d5d805c4b5cdd1b2f16d3040351b9fdf7d9b8d309d10 not found: ID does not exist" Apr 23 19:13:55.549287 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.549218 2575 scope.go:117] "RemoveContainer" containerID="7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367" Apr 23 19:13:55.549452 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.549432 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367"} err="failed to get container status \"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367\": rpc error: code = NotFound desc = could not find container \"7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367\": container with ID starting with 7621fccfe1e25ad740c04947d62d30f9e30c70dd607a1b44e851143f8944d367 not found: ID does not exist" Apr 23 19:13:55.549452 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.549450 2575 scope.go:117] "RemoveContainer" containerID="a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209" Apr 23 19:13:55.556512 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.556490 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg"] Apr 23 19:13:55.558072 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.558058 2575 scope.go:117] "RemoveContainer" containerID="186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a" Apr 23 19:13:55.561372 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.561349 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f974f4c44-cwjhg"] Apr 23 19:13:55.595195 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.595159 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698575f0-3e94-40e2-962f-8398eac7549a" path="/var/lib/kubelet/pods/698575f0-3e94-40e2-962f-8398eac7549a/volumes" Apr 23 19:13:55.595642 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.595628 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" path="/var/lib/kubelet/pods/8739b531-9c81-469e-8f3c-3b6682d00937/volumes" Apr 23 19:13:55.622552 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.622530 2575 scope.go:117] "RemoveContainer" containerID="a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209" Apr 23 19:13:55.622891 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:13:55.622870 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209\": container with ID starting with a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209 not found: ID does not exist" containerID="a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209" Apr 23 19:13:55.622965 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.622902 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209"} err="failed to get container status \"a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209\": rpc error: code = NotFound desc = could not find container \"a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209\": container with ID starting with a8de9c17e11caa3d5a2fba2d01bafc1ebdccc7983078ea69dc66e879fd33b209 not found: ID does not exist" Apr 23 19:13:55.622965 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.622924 2575 scope.go:117] "RemoveContainer" containerID="186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a" Apr 23 19:13:55.623181 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:13:55.623161 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a\": container with ID starting with 186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a not found: ID does not exist" containerID="186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a" Apr 23 19:13:55.623218 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:13:55.623188 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a"} err="failed to get container status \"186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a\": rpc error: code = NotFound desc = could not find container \"186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a\": container with ID starting with 186b5434d0fdc1b1115f9debef38b67cd3bfa74ca69c5f9792274b5f78be983a not found: ID does not exist" Apr 23 19:16:21.046164 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046128 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8n7dv/must-gather-cqx24"] Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046457 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046468 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046477 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046482 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046491 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046497 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046504 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046515 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046520 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046527 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046532 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046539 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="llm-d-routing-sidecar" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046544 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="llm-d-routing-sidecar" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046553 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046557 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046564 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="storage-initializer" Apr 23 19:16:21.046623 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046568 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="storage-initializer" Apr 23 19:16:21.047162 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046642 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="llm-d-routing-sidecar" Apr 23 19:16:21.047162 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046651 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="8739b531-9c81-469e-8f3c-3b6682d00937" containerName="main" Apr 23 19:16:21.047162 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046657 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8ea5617-3af1-43f2-8d98-2aeb8e37526d" containerName="main" Apr 23 19:16:21.047162 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046665 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="698575f0-3e94-40e2-962f-8398eac7549a" containerName="main" Apr 23 19:16:21.047162 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.046671 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="7526c47a-1e88-49a7-b7b7-b6bb50e80fd8" containerName="main" Apr 23 19:16:21.049216 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.049199 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.053181 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.053155 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8n7dv\"/\"openshift-service-ca.crt\"" Apr 23 19:16:21.053332 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.053178 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8n7dv\"/\"kube-root-ca.crt\"" Apr 23 19:16:21.053332 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.053159 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8n7dv\"/\"default-dockercfg-4vwgt\"" Apr 23 19:16:21.060404 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.060377 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8n7dv/must-gather-cqx24"] Apr 23 19:16:21.067846 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.067821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-must-gather-output\") pod \"must-gather-cqx24\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.067977 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.067892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rrqd\" (UniqueName: \"kubernetes.io/projected/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-kube-api-access-5rrqd\") pod \"must-gather-cqx24\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.168564 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.168525 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-must-gather-output\") pod \"must-gather-cqx24\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.168765 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.168626 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5rrqd\" (UniqueName: \"kubernetes.io/projected/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-kube-api-access-5rrqd\") pod \"must-gather-cqx24\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.168923 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.168902 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-must-gather-output\") pod \"must-gather-cqx24\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.180895 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.180859 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rrqd\" (UniqueName: \"kubernetes.io/projected/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-kube-api-access-5rrqd\") pod \"must-gather-cqx24\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.359263 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.359170 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:21.487566 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.487534 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8n7dv/must-gather-cqx24"] Apr 23 19:16:21.491754 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:16:21.491724 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ce60a28_f126_4c3e_aeb4_f9f208791ef8.slice/crio-89e4d4c465c0b186edff76547a26d553a4a5dc4ce7f1c95bcc5c497f652189c4 WatchSource:0}: Error finding container 89e4d4c465c0b186edff76547a26d553a4a5dc4ce7f1c95bcc5c497f652189c4: Status 404 returned error can't find the container with id 89e4d4c465c0b186edff76547a26d553a4a5dc4ce7f1c95bcc5c497f652189c4 Apr 23 19:16:21.493325 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:21.493307 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 23 19:16:22.011089 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:22.011053 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8n7dv/must-gather-cqx24" event={"ID":"4ce60a28-f126-4c3e-aeb4-f9f208791ef8","Type":"ContainerStarted","Data":"89e4d4c465c0b186edff76547a26d553a4a5dc4ce7f1c95bcc5c497f652189c4"} Apr 23 19:16:26.027980 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:26.027943 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8n7dv/must-gather-cqx24" event={"ID":"4ce60a28-f126-4c3e-aeb4-f9f208791ef8","Type":"ContainerStarted","Data":"4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e"} Apr 23 19:16:26.027980 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:26.027983 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8n7dv/must-gather-cqx24" event={"ID":"4ce60a28-f126-4c3e-aeb4-f9f208791ef8","Type":"ContainerStarted","Data":"5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4"} Apr 23 19:16:26.047804 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:26.047756 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-8n7dv/must-gather-cqx24" podStartSLOduration=1.003559847 podStartE2EDuration="5.047740864s" podCreationTimestamp="2026-04-23 19:16:21 +0000 UTC" firstStartedPulling="2026-04-23 19:16:21.493431946 +0000 UTC m=+5680.512884944" lastFinishedPulling="2026-04-23 19:16:25.537612962 +0000 UTC m=+5684.557065961" observedRunningTime="2026-04-23 19:16:26.046737348 +0000 UTC m=+5685.066190364" watchObservedRunningTime="2026-04-23 19:16:26.047740864 +0000 UTC m=+5685.067193879" Apr 23 19:16:41.864012 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:41.863910 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:16:41.878637 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:41.873617 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:16:50.355926 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:50.355892 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-747f569c94-mjbpd_2dc66a6c-a16e-41b1-9fd0-cd602ca9e666/router/0.log" Apr 23 19:16:51.245996 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:51.245965 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-747f569c94-mjbpd_2dc66a6c-a16e-41b1-9fd0-cd602ca9e666/router/0.log" Apr 23 19:16:52.076684 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:52.076643 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-vlxjh_a7814e01-e9c6-44f6-996c-a7514e3ab922/authorino/0.log" Apr 23 19:16:52.108476 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:52.108445 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-hhjsv_a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b/manager/0.log" Apr 23 19:16:52.123992 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:52.123969 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-mj4k9_a0c0f2c4-353c-4741-81a8-57f026b023d9/kuadrant-console-plugin/0.log" Apr 23 19:16:52.220138 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:52.220109 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-52nnx_903d7de4-4f05-4999-bc57-9700e74f64b3/limitador/0.log" Apr 23 19:16:53.131108 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:53.131073 2575 generic.go:358] "Generic (PLEG): container finished" podID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerID="5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4" exitCode=0 Apr 23 19:16:53.131506 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:53.131143 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8n7dv/must-gather-cqx24" event={"ID":"4ce60a28-f126-4c3e-aeb4-f9f208791ef8","Type":"ContainerDied","Data":"5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4"} Apr 23 19:16:53.131506 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:53.131435 2575 scope.go:117] "RemoveContainer" containerID="5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4" Apr 23 19:16:53.925827 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:53.925791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8n7dv_must-gather-cqx24_4ce60a28-f126-4c3e-aeb4-f9f208791ef8/gather/0.log" Apr 23 19:16:57.709644 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:57.709606 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hhn8r_47d84ec3-de06-48f4-bafd-89def58eb514/global-pull-secret-syncer/0.log" Apr 23 19:16:57.815523 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:57.815495 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bxxbs_95f1b7b2-68c2-4b1d-8939-d348b6b5daf0/konnectivity-agent/0.log" Apr 23 19:16:57.949753 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:57.949723 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-68.ec2.internal_b84d58d2271f831c3e7ac872fa85f304/haproxy/0.log" Apr 23 19:16:59.434208 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.434170 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8n7dv/must-gather-cqx24"] Apr 23 19:16:59.434811 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.434404 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-8n7dv/must-gather-cqx24" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="copy" containerID="cri-o://4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e" gracePeriod=2 Apr 23 19:16:59.436740 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.436707 2575 status_manager.go:895] "Failed to get status for pod" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" pod="openshift-must-gather-8n7dv/must-gather-cqx24" err="pods \"must-gather-cqx24\" is forbidden: User \"system:node:ip-10-0-137-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8n7dv\": no relationship found between node 'ip-10-0-137-68.ec2.internal' and this object" Apr 23 19:16:59.437116 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.437089 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8n7dv/must-gather-cqx24"] Apr 23 19:16:59.666454 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.666429 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8n7dv_must-gather-cqx24_4ce60a28-f126-4c3e-aeb4-f9f208791ef8/copy/0.log" Apr 23 19:16:59.666788 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.666772 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:16:59.779999 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.779965 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-must-gather-output\") pod \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " Apr 23 19:16:59.780147 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.780033 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rrqd\" (UniqueName: \"kubernetes.io/projected/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-kube-api-access-5rrqd\") pod \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\" (UID: \"4ce60a28-f126-4c3e-aeb4-f9f208791ef8\") " Apr 23 19:16:59.782200 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.782165 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-kube-api-access-5rrqd" (OuterVolumeSpecName: "kube-api-access-5rrqd") pod "4ce60a28-f126-4c3e-aeb4-f9f208791ef8" (UID: "4ce60a28-f126-4c3e-aeb4-f9f208791ef8"). InnerVolumeSpecName "kube-api-access-5rrqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 23 19:16:59.786169 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.786148 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4ce60a28-f126-4c3e-aeb4-f9f208791ef8" (UID: "4ce60a28-f126-4c3e-aeb4-f9f208791ef8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 23 19:16:59.881027 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.880997 2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-must-gather-output\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:16:59.881027 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:16:59.881025 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rrqd\" (UniqueName: \"kubernetes.io/projected/4ce60a28-f126-4c3e-aeb4-f9f208791ef8-kube-api-access-5rrqd\") on node \"ip-10-0-137-68.ec2.internal\" DevicePath \"\"" Apr 23 19:17:00.154730 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.154653 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8n7dv_must-gather-cqx24_4ce60a28-f126-4c3e-aeb4-f9f208791ef8/copy/0.log" Apr 23 19:17:00.155006 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.154984 2575 generic.go:358] "Generic (PLEG): container finished" podID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerID="4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e" exitCode=143 Apr 23 19:17:00.155066 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.155035 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8n7dv/must-gather-cqx24" Apr 23 19:17:00.155107 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.155085 2575 scope.go:117] "RemoveContainer" containerID="4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e" Apr 23 19:17:00.162757 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.162737 2575 scope.go:117] "RemoveContainer" containerID="5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4" Apr 23 19:17:00.174343 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.174326 2575 scope.go:117] "RemoveContainer" containerID="4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e" Apr 23 19:17:00.174578 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:17:00.174556 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e\": container with ID starting with 4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e not found: ID does not exist" containerID="4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e" Apr 23 19:17:00.174679 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.174605 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e"} err="failed to get container status \"4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e\": rpc error: code = NotFound desc = could not find container \"4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e\": container with ID starting with 4fd8b0b35b99a550d3c7eefa04f8dcd265dd235265de2fa9408a274606db0e1e not found: ID does not exist" Apr 23 19:17:00.174679 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.174633 2575 scope.go:117] "RemoveContainer" containerID="5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4" Apr 23 19:17:00.174857 ip-10-0-137-68 kubenswrapper[2575]: E0423 19:17:00.174840 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4\": container with ID starting with 5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4 not found: ID does not exist" containerID="5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4" Apr 23 19:17:00.174898 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:00.174864 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4"} err="failed to get container status \"5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4\": rpc error: code = NotFound desc = could not find container \"5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4\": container with ID starting with 5e6df2db31cd35c270d4f6923b7bf4f2a93309cb2ac868bee8850e7d78e7bce4 not found: ID does not exist" Apr 23 19:17:01.597078 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:01.597042 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" path="/var/lib/kubelet/pods/4ce60a28-f126-4c3e-aeb4-f9f208791ef8/volumes" Apr 23 19:17:01.612445 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:01.612419 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-vlxjh_a7814e01-e9c6-44f6-996c-a7514e3ab922/authorino/0.log" Apr 23 19:17:01.682042 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:01.682007 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-hhjsv_a46f9ddd-b4c3-45e1-9887-9e9fd258cd6b/manager/0.log" Apr 23 19:17:01.710777 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:01.710729 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-mj4k9_a0c0f2c4-353c-4741-81a8-57f026b023d9/kuadrant-console-plugin/0.log" Apr 23 19:17:01.791051 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:01.790970 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-52nnx_903d7de4-4f05-4999-bc57-9700e74f64b3/limitador/0.log" Apr 23 19:17:02.957256 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:02.957225 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/alertmanager/0.log" Apr 23 19:17:02.986253 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:02.986232 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/config-reloader/0.log" Apr 23 19:17:03.013477 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.013448 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/kube-rbac-proxy-web/0.log" Apr 23 19:17:03.041356 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.041331 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/kube-rbac-proxy/0.log" Apr 23 19:17:03.068689 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.068662 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/kube-rbac-proxy-metric/0.log" Apr 23 19:17:03.101521 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.101492 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/prom-label-proxy/0.log" Apr 23 19:17:03.133150 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.133116 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e838daf-b1b9-4d33-a33b-8cf40f382378/init-config-reloader/0.log" Apr 23 19:17:03.199664 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.199631 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-75587bd455-5cln8_eb0a83a2-e27a-459f-a491-f2ffb4073120/cluster-monitoring-operator/0.log" Apr 23 19:17:03.366052 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.366024 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jj65_7a209846-480f-458a-94ef-f0de3126b8a5/node-exporter/0.log" Apr 23 19:17:03.388704 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.388669 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jj65_7a209846-480f-458a-94ef-f0de3126b8a5/kube-rbac-proxy/0.log" Apr 23 19:17:03.412733 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:03.412711 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jj65_7a209846-480f-458a-94ef-f0de3126b8a5/init-textfile/0.log" Apr 23 19:17:06.195979 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.195949 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv"] Apr 23 19:17:06.196341 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.196264 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="gather" Apr 23 19:17:06.196341 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.196274 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="gather" Apr 23 19:17:06.196341 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.196284 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="copy" Apr 23 19:17:06.196341 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.196290 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="copy" Apr 23 19:17:06.196341 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.196335 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="copy" Apr 23 19:17:06.196505 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.196346 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ce60a28-f126-4c3e-aeb4-f9f208791ef8" containerName="gather" Apr 23 19:17:06.199203 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.199187 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.201501 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.201478 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j4sk6\"/\"default-dockercfg-tvs57\"" Apr 23 19:17:06.201707 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.201694 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j4sk6\"/\"openshift-service-ca.crt\"" Apr 23 19:17:06.202365 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.202348 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j4sk6\"/\"kube-root-ca.crt\"" Apr 23 19:17:06.208927 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.208905 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv"] Apr 23 19:17:06.335709 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.335670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-podres\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.335709 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.335712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-proc\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.335935 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.335755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-lib-modules\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.335935 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.335780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-sys\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.335935 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.335799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldnh\" (UniqueName: \"kubernetes.io/projected/41e17569-2677-4bbc-83a6-3abe6d9014a5-kube-api-access-6ldnh\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.436944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.436909 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-podres\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.436944 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.436944 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-proc\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.436979 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-lib-modules\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.436998 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-sys\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.437018 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldnh\" (UniqueName: \"kubernetes.io/projected/41e17569-2677-4bbc-83a6-3abe6d9014a5-kube-api-access-6ldnh\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.437083 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-podres\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.437101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-proc\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.437137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-lib-modules\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.437187 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.437143 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e17569-2677-4bbc-83a6-3abe6d9014a5-sys\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.445719 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.445696 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldnh\" (UniqueName: \"kubernetes.io/projected/41e17569-2677-4bbc-83a6-3abe6d9014a5-kube-api-access-6ldnh\") pod \"perf-node-gather-daemonset-k9qmv\" (UID: \"41e17569-2677-4bbc-83a6-3abe6d9014a5\") " pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.509513 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.509486 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:06.539195 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.539166 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b498c6d49-nlgtd_6fcd778e-9bf3-428c-8028-600482b5ef44/console/0.log" Apr 23 19:17:06.638668 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:06.638638 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv"] Apr 23 19:17:06.640763 ip-10-0-137-68 kubenswrapper[2575]: W0423 19:17:06.640727 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod41e17569_2677_4bbc_83a6_3abe6d9014a5.slice/crio-18efcf30b2a8d5ec5227b786af0ad1dbef1015becfdd8e50b2a0d5c8d6a28a49 WatchSource:0}: Error finding container 18efcf30b2a8d5ec5227b786af0ad1dbef1015becfdd8e50b2a0d5c8d6a28a49: Status 404 returned error can't find the container with id 18efcf30b2a8d5ec5227b786af0ad1dbef1015becfdd8e50b2a0d5c8d6a28a49 Apr 23 19:17:07.049484 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.049450 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-7c6cbb6c87-qgzxg_5c365186-3cfd-4cd1-868f-72da2be8292b/volume-data-source-validator/0.log" Apr 23 19:17:07.178952 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.178916 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" event={"ID":"41e17569-2677-4bbc-83a6-3abe6d9014a5","Type":"ContainerStarted","Data":"564d6daf099b9f1605cabcd1556173f6e4ef640ac0f9711956270368f2d70f9c"} Apr 23 19:17:07.178952 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.178955 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" event={"ID":"41e17569-2677-4bbc-83a6-3abe6d9014a5","Type":"ContainerStarted","Data":"18efcf30b2a8d5ec5227b786af0ad1dbef1015becfdd8e50b2a0d5c8d6a28a49"} Apr 23 19:17:07.179164 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.178984 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:07.196895 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.196611 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" podStartSLOduration=1.19657648 podStartE2EDuration="1.19657648s" podCreationTimestamp="2026-04-23 19:17:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-23 19:17:07.196487755 +0000 UTC m=+5726.215940782" watchObservedRunningTime="2026-04-23 19:17:07.19657648 +0000 UTC m=+5726.216029480" Apr 23 19:17:07.848933 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.848903 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9786k_3459e220-189c-4bb2-91fd-ebd60ad6899e/dns/0.log" Apr 23 19:17:07.873203 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:07.873179 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9786k_3459e220-189c-4bb2-91fd-ebd60ad6899e/kube-rbac-proxy/0.log" Apr 23 19:17:08.000136 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:08.000103 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-728bd_bb810179-e96c-4f26-a7fc-e3006013faed/dns-node-resolver/0.log" Apr 23 19:17:08.698095 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:08.698060 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xlkpg_9a5b8fa2-c8ef-4328-b02c-5663e2246f76/node-ca/0.log" Apr 23 19:17:09.711217 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:09.711188 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-747f569c94-mjbpd_2dc66a6c-a16e-41b1-9fd0-cd602ca9e666/router/0.log" Apr 23 19:17:10.225547 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:10.225511 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bwrpd_d2b26376-6f53-4edd-89f2-a4fe64523424/serve-healthcheck-canary/0.log" Apr 23 19:17:10.907707 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:10.907675 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wsvkb_0a44b595-de5f-4334-bb31-1c5aa5d798a2/kube-rbac-proxy/0.log" Apr 23 19:17:10.931172 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:10.931143 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wsvkb_0a44b595-de5f-4334-bb31-1c5aa5d798a2/exporter/0.log" Apr 23 19:17:10.954855 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:10.954829 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wsvkb_0a44b595-de5f-4334-bb31-1c5aa5d798a2/extractor/0.log" Apr 23 19:17:13.191865 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:13.191835 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j4sk6/perf-node-gather-daemonset-k9qmv" Apr 23 19:17:13.627092 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:13.627061 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bc7774745-kjk2m_b132adb9-73b5-43bd-b408-9d6da8e3ba4a/manager/0.log" Apr 23 19:17:13.651731 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:13.651697 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-wxdcx_eb4957a3-2ef0-4ac0-ba04-2a568e4a4f46/openshift-lws-operator/0.log" Apr 23 19:17:19.530334 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:19.530297 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-74bb7799d9-d6bzf_83b95dec-6e53-4691-bce3-2f6d74fabb90/migrator/0.log" Apr 23 19:17:19.554747 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:19.554718 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-74bb7799d9-d6bzf_83b95dec-6e53-4691-bce3-2f6d74fabb90/graceful-termination/0.log" Apr 23 19:17:21.305313 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.305283 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/kube-multus-additional-cni-plugins/0.log" Apr 23 19:17:21.334976 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.334952 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/egress-router-binary-copy/0.log" Apr 23 19:17:21.361338 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.361310 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/cni-plugins/0.log" Apr 23 19:17:21.395546 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.395518 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/bond-cni-plugin/0.log" Apr 23 19:17:21.419035 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.419010 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/routeoverride-cni/0.log" Apr 23 19:17:21.445933 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.445910 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/whereabouts-cni-bincopy/0.log" Apr 23 19:17:21.472096 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.472069 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7svv_7f9db399-293d-460a-8a3e-294b3c70c30e/whereabouts-cni/0.log" Apr 23 19:17:21.746076 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.746045 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lvm6f_97c7e75c-b4d5-4eef-b1fd-a9102c87fccc/kube-multus/0.log" Apr 23 19:17:21.770091 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.770054 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-76492_eb1ab24a-5f1e-4324-9b5f-f92abb7647fa/network-metrics-daemon/0.log" Apr 23 19:17:21.792227 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:21.792201 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-76492_eb1ab24a-5f1e-4324-9b5f-f92abb7647fa/kube-rbac-proxy/0.log" Apr 23 19:17:23.060511 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.060479 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-controller/0.log" Apr 23 19:17:23.082785 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.082761 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/0.log" Apr 23 19:17:23.131228 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.131190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovn-acl-logging/1.log" Apr 23 19:17:23.161873 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.161845 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/kube-rbac-proxy-node/0.log" Apr 23 19:17:23.187192 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.187160 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/kube-rbac-proxy-ovn-metrics/0.log" Apr 23 19:17:23.206807 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.206786 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/northd/0.log" Apr 23 19:17:23.235237 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.235213 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/nbdb/0.log" Apr 23 19:17:23.256839 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.256770 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/sbdb/0.log" Apr 23 19:17:23.430465 ip-10-0-137-68 kubenswrapper[2575]: I0423 19:17:23.430437 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7h4b_f3973fd7-657b-4408-87a3-48d5d15b2e21/ovnkube-controller/0.log"