Sep 05 02:16:13.481946 ip-10-0-140-68 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 05 02:16:13.481963 ip-10-0-140-68 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 05 02:16:13.481972 ip-10-0-140-68 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 05 02:16:13.482414 ip-10-0-140-68 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 05 02:16:23.571310 ip-10-0-140-68 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 05 02:16:23.571331 ip-10-0-140-68 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 80d1ccb485f647c483edb2e451a1b35f --
Sep 05 02:18:19.072222 ip-10-0-140-68 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:18:19.426855 ip-10-0-140-68 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:19.426855 ip-10-0-140-68 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:18:19.426855 ip-10-0-140-68 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:19.426855 ip-10-0-140-68 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:18:19.426855 ip-10-0-140-68 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:19.429019 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.428932    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:18:19.432701 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432686    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:19.432701 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432701    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432705    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432708    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432711    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432727    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432731    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432734    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432737    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432741    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432744    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432746    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432749    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432751    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432754    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432756    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432759    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432761    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432764    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432766    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432768    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:19.432773 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432772    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432775    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432778    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432780    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432783    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432786    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432789    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432791    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432794    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432796    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432799    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432803    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432806    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432809    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432812    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432815    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432818    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432820    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432823    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:19.433259 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432826    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432830    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432833    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432836    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432838    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432840    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432843    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432845    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432847    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432857    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432859    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432862    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432864    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432867    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432870    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432873    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432876    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432879    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432881    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:19.433713 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432884    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432886    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432889    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432891    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432894    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432897    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432899    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432902    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432904    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432907    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432909    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432911    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432915    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432917    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432920    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432922    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432924    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432930    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432933    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432936    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:19.434224 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432953    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432957    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432959    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432962    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432965    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432968    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432971    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432974    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432977    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432980    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432983    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432986    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432989    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432991    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432993    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432996    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.432999    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433001    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433004    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433006    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:19.434712 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433679    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433686    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433690    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433694    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433697    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433700    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433702    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433716    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433720    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433724    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433727    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433730    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433733    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433735    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433738    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433740    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433743    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433746    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433749    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:19.435201 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433751    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433755    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433758    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433761    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433763    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433766    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433768    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433771    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433773    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433776    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433778    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433780    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433783    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433785    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433788    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433790    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433793    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433795    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433797    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433801    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:19.435669 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433805    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433808    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433811    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433814    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433816    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433819    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433821    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433823    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433826    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433828    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433831    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433835    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433837    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433840    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433842    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433845    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433847    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433850    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433852    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433855    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:19.436278 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433858    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433860    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433862    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433865    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433868    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433870    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433874    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433877    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433880    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433883    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433886    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433889    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433892    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433894    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433897    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433900    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433902    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433905    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433908    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:19.436767 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433911    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433914    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433917    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433920    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433922    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433925    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433928    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433930    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433932    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433935    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433953    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433956    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433959    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433961    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433964    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433966    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433969    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433972    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433974    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433977    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:19.437250 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.433979    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434705    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434711    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434714    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434718    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434723    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434726    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434729    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434733    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434736    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434738    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434741    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434744    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434746    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434749    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434752    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434754    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434757    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434759    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:19.437732 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434762    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434765    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434767    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434770    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434773    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434775    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434777    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434780    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434782    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434785    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434787    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434790    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434793    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434795    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434798    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434800    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434803    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434805    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434808    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:19.438215 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434812    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434815    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434817    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434820    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434822    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434825    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434828    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434830    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434834    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434837    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434839    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434842    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434845    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434847    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434849    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434852    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434855    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434858    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434860    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434863    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:19.438702 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434865    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434867    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434870    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434872    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434874    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434877    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434879    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434882    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434884    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434886    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434891    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434894    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434897    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434900    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434904    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434907    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434909    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434912    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434914    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:19.439197 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434917    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434920    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434923    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434925    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434928    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434930    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434933    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434935    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434953    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434955    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434958    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434961    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434964    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434966    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434969    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434971    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434974    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434976    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434979    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434981    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:19.439650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434984    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434986    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.434989    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435065    2568 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435103    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435109    2568 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435113    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435119    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435122    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435126    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435130    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435133    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435136    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435140    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435144    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435146    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435149    2568 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435152    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435155    2568 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435158    2568 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435161    2568 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435165    2568 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435167    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435171    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:18:19.440165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435174    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435177    2568 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435181    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435184    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435187    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435190    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435193    2568 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435195    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435198    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435201    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435204    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435208    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435211    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435214    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435217    2568 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435220    2568 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435223    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435227    2568 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435230    2568 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435232    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435235    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435240    2568 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435244    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435246    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435249    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:18:19.440765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435252    2568 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435254    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435257    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435260    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435263    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435266    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435268    2568 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435272    2568 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435276    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435279    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435282    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435284    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435287    2568 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435290    2568 flags.go:64] FLAG: --help="false"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435293    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435296    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435299    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435302    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435305    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435308    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435311    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435314    2568 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435317    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:18:19.441409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435320    2568 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435323    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435326    2568 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435328    2568 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435331    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435335    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435338    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435341    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435343    2568 flags.go:64] FLAG: --lock-file=""
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435346    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435349    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435352    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435357    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435360    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435362    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435366    2568 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435368    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435372    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435375    2568 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435377    2568 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435381    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435384    2568 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435388    2568 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435391    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435394    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:18:19.442366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435396    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435399    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435402    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435405    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435408    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435414    2568 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435417    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435420    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435423    2568 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435425    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435431    2568 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435433    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435436    2568 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435442    2568 flags.go:64] FLAG: --port="10250"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435445    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435448    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b31c3b315a6e7703"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435451    2568 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435453    2568 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435456    2568 flags.go:64] FLAG: --register-node="true"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435459    2568 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435462    2568 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435465    2568 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435468    2568 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435471    2568 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435475    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:18:19.443295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435478    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435481    2568 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435484    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435487    2568 flags.go:64] FLAG: --runonce="false"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435489    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435493    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435495    2568 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435498    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435501    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435504    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435507    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435509    2568 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435512    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435515    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435517    2568 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435520    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435523    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435526    2568 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435529    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435533    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435537    2568 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435540    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435543    2568 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435546    2568 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435548    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:18:19.443921 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435551    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435554    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435556    2568 flags.go:64] FLAG: --v="2"
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435560    2568 flags.go:64] FLAG: --version="false"
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435564    2568 flags.go:64] FLAG: --vmodule=""
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435567    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435570    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435673    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435677    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435680    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435683    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435685    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435689    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435693    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435696    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435698    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435701    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435703    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435707    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435709    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435712    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:19.444543 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435714    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435717    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435719    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435722    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435724    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435727    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435731    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435733    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435736    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435739    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435741    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435743    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435746    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435748    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435751    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435754    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435756    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435758    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435761    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:19.445176 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435763    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435766    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435769    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435771    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435774    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435776    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435778    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435781    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435783    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435785    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435790    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435792    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435794    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435797    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435799    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435802    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435804    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435807    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435810    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435814    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:19.445642 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435816    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435827    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435830    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435833    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435835    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435838    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435840    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435843    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435845    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435848    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435852    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435856    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435859    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435861    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435864    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435867    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435869    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435872    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435874    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:19.446156 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435877    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435879    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435882    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435887    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435889    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435892    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435895    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435897    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435900    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435902    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435905    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435907    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435911    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435913    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435916    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435918    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435921    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435923    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435926    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435929    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:19.446660 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435931    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435934    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435951    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435954    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435957    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435960    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.435963    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.435969    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.444255    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.444270    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.444294    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.444303    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444370    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444376    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:19.447199 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444380    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444385    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444388    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444391    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444395    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444399    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444401    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444404    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444407    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444409    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444412    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444415    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444417    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444420    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444423    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444425    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444428    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444431    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444433    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:19.447566 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444436    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444438    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444441    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444443    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444446    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444448    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444451    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444453    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444457    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444460    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444462    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444465    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444467    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444470    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444473    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444475    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444477    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444480    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444483    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444485    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:19.448098 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444488    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444490    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444493    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444495    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444498    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444508    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444511    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444514    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444516    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444519    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444522    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444526    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444529    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444532    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444535    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444538    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444540    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444543    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444545    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:19.448674 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444548    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444552    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444554    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444557    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444560    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444562    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444565    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444568    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444570    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444573    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444575    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444578    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444580    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444582    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444585    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444587    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444590    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444592    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444595    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444603    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:19.449264 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444606    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444608    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444611    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444613    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444616    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444619    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444621    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444624    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444626    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444629    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444632    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444635    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444638    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444640    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444643    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444646    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444648    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444651    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:19.450159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444653    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.444658    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444767    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444772    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444775    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444777    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444781    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444784    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444787    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444790    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444793    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444795    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444798    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444801    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444810    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:19.450650 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444812    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444815    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444818    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444820    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444823    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444826    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444828    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444831    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444834    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444838    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444840    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444843    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444845    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444848    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444850    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444853    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444855    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444858    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444860    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444862    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:19.451047 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444865    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444867    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444872    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444875    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444878    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444880    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444883    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444886    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444889    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444891    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444894    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444896    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444900    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444902    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444904    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444907    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444910    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444913    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444915    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:19.451572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444918    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444920    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444923    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444925    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444928    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444931    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444933    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444936    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444958    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444962    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444964    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444967    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444970    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444972    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444975    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444977    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444980    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444982    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444985    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:19.452085 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444987    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444990    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444992    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444995    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444997    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.444999    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445002    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445006    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445009    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445011    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445014    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445016    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445019    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445021    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445024    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445026    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445029    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445031    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445034    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445036    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:19.452584 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445038    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445041    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445043    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445046    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445048    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445050    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445053    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:19.445057    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.445062    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.445163    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.446960    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.447907    2568 server.go:1019] "Starting client certificate rotation"
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.447999    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:18:19.453387 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.448042    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:18:19.453706 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.450180    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:18:19.454220 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.454206    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:18:19.463225 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.463208    2568 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:18:19.465794 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.465778    2568 log.go:25] "Validated CRI v1 image API"
Sep 05 02:18:19.466733 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.466705    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:18:19.471898 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.471879    2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ce9981c5-a1f6-436b-b012-b96f07cec185:/dev/nvme0n1p4 ecf4a6e3-332f-4273-ac3b-d4313e532721:/dev/nvme0n1p3]
Sep 05 02:18:19.471985 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.471897    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:18:19.478763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.478630    2568 manager.go:216] Machine: {Timestamp:2026-09-05 02:18:19.476557318 +0000 UTC m=+0.298804698 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101017 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b47e61f4a867349ed74abc19268b9 SystemUUID:ec2b47e6-1f4a-8673-49ed-74abc19268b9 BootID:80d1ccb4-85f6-47c4-83ed-b2e451a1b35f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ec:5e:79:d3:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ec:5e:79:d3:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:42:a5:12:b8:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:18:19.478763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.478758    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:18:19.478894 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.478835    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:18:19.481613 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.481590    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:18:19.481749 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.481616    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:18:19.482297 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.482280    2568 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:18:19.482350 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.482301    2568 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:18:19.482350 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.482311    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:18:19.482350 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.482345    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:18:19.483080 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.483070    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:18:19.485542 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485527    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:19.485687 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485676    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:18:19.485831 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485821    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:18:19.485865 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485849    2568 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:18:19.485892 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485865    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:18:19.485935 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485927    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:18:19.485980 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485965    2568 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:18:19.485980 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.485979    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:18:19.487145 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.487124    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:18:19.487247 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.487203    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:18:19.490915 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.490888    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:18:19.492776 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.492762    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:18:19.492818 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.492793    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:18:19.493573 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493562    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:18:19.493614 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493584    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:18:19.493614 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493590    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:18:19.493614 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493595    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:18:19.493614 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493603    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:18:19.493614 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493608    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:18:19.493759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493621    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:18:19.493759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493627    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:18:19.493759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493637    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:18:19.493759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493644    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:18:19.493759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493653    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:18:19.493759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.493658    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:18:19.494828 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.494818    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:18:19.494828 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.494828    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:18:19.499074 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.498917    2568 server.go:1268] "Started kubelet"
Sep 05 02:18:19.499184 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.499028    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:18:19.499705 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.499683    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:18:19.499781 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.499736    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:18:19.499998 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.499917    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:18:19.500287 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.500209    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:18:19.500364 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.500301    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:18:19.500594 ip-10-0-140-68 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:18:19.501769 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.501753    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:18:19.502233 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.502209    2568 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:18:19.503109 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.503092    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:18:19.504437 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.504422    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:19.504846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.504833    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:18:19.504916 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.504900    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:19.505274 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505255    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:18:19.505274 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505276    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:18:19.505420 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505372    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:18:19.505420 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505384    2568 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:18:19.505420 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505395    2568 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:18:19.505574 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505558    2568 factory.go:55] Registering systemd factory
Sep 05 02:18:19.505574 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505572    2568 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:18:19.505821 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505808    2568 factory.go:153] Registering CRI-O factory
Sep 05 02:18:19.505860 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505826    2568 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:18:19.505895 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505872    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:18:19.505895 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505891    2568 factory.go:103] Registering Raw factory
Sep 05 02:18:19.505998 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.505903    2568 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:18:19.506193 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.505118    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.ec2.internal.18d24b3681e1d6ec  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.ec2.internal,UID:ip-10-0-140-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.ec2.internal,},FirstTimestamp:2026-09-05 02:18:19.498886892 +0000 UTC m=+0.321134272,LastTimestamp:2026-09-05 02:18:19.498886892 +0000 UTC m=+0.321134272,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.ec2.internal,}"
Sep 05 02:18:19.506708 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.506697    2568 manager.go:349] Starting recovery of all containers
Sep 05 02:18:19.513073 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.512503    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qj5x8"
Sep 05 02:18:19.515110 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.515083    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 02:18:19.515231 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.515117    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 02:18:19.519504 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.519488    2568 manager.go:354] Recovery completed
Sep 05 02:18:19.521433 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.521322    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qj5x8"
Sep 05 02:18:19.523749 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.523735    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.525523 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.525505    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.525610 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.525536    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.525610 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.525547    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.526165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526153    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:18:19.526165 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526164    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:18:19.526256 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526178    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:18:19.526256 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526184    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.526899 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526885    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.526984 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526916    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.526984 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.526930    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.527163 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.527150    2568 policy_none.go:49] "None policy: Start"
Sep 05 02:18:19.527205 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.527165    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:18:19.527205 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.527176    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:18:19.527661 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.527598    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.ec2.internal.18d24b36837843e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.ec2.internal,UID:ip-10-0-140-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-68.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.ec2.internal,},FirstTimestamp:2026-09-05 02:18:19.525522405 +0000 UTC m=+0.347769790,LastTimestamp:2026-09-05 02:18:19.525522405 +0000 UTC m=+0.347769790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.ec2.internal,}"
Sep 05 02:18:19.528010 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.528000    2568 policy_none.go:47] "Start"
Sep 05 02:18:19.562534 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.562518    2568 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.562555    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.562568    2568 server.go:87] "Starting device plugin registration server"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.562812    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.562823    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.562919    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.563010    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.563018    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.563483    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:18:19.575249 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.563530    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:19.628011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.627982    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:18:19.629238 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.629215    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:18:19.629316 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.629269    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:18:19.629316 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.629295    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:18:19.629422 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.629337    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:18:19.631973 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.631954    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:19.663183 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.663164    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.663904 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.663890    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.663979 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.663919    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.663979 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.663929    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.663979 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.663969    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.670298 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.670285    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.670339 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.670303    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-68.ec2.internal\": node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:19.687294 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.687240    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:19.729414 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.729380    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"]
Sep 05 02:18:19.729469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.729454    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.730206 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.730193    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.730274 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.730219    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.730274 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.730234    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.732610 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.732598    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.732730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.732718    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.732763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.732744    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.733255 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.733238    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.733347 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.733270    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.733347 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.733239    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.733347 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.733280    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.733347 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.733294    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.733347 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.733307    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.735463 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.735450    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.735519 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.735473    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:19.736202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.736186    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:19.736278 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.736216    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:19.736278 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.736232    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:19.748438 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.748421    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.ec2.internal\" not found" node="ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.752105 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.752092    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.ec2.internal\" not found" node="ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.788325 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.788305    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:19.807180 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.807161    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bf1afb938f794d571614842ff701595-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"5bf1afb938f794d571614842ff701595\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.807241 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.807184    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf1afb938f794d571614842ff701595-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"5bf1afb938f794d571614842ff701595\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.807241 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.807199    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63872a04a76bb538711c93178fe3d471-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.ec2.internal\" (UID: \"63872a04a76bb538711c93178fe3d471\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.888775 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.888742    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:19.908099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.908079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bf1afb938f794d571614842ff701595-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"5bf1afb938f794d571614842ff701595\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.908169 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.908109    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf1afb938f794d571614842ff701595-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"5bf1afb938f794d571614842ff701595\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.908206 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.908177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63872a04a76bb538711c93178fe3d471-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.ec2.internal\" (UID: \"63872a04a76bb538711c93178fe3d471\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.908239 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.908213    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bf1afb938f794d571614842ff701595-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"5bf1afb938f794d571614842ff701595\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.908268 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.908223    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63872a04a76bb538711c93178fe3d471-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.ec2.internal\" (UID: \"63872a04a76bb538711c93178fe3d471\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.908298 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:19.908270    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf1afb938f794d571614842ff701595-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal\" (UID: \"5bf1afb938f794d571614842ff701595\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:19.989375 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:19.989353    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.049735 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.049711    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:20.054860 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.054827    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:20.090201 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.090174    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.190760 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.190713    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.291224 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.291155    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.391726 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.391700    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.448199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.448177    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:18:20.448617 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.448324    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:18:20.492792 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.492753    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.520736 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.520711    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:20.524643 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.524619    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:13:20 +0000 UTC" deadline="2028-05-29 03:39:32.690224746 +0000 UTC"
Sep 05 02:18:20.524706 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.524644    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15169h21m12.165584044s"
Sep 05 02:18:20.580936 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:20.580910    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63872a04a76bb538711c93178fe3d471.slice/crio-2d5ce33b9f5e50f324b0ba1d91768db58788a5ad5dedb285fd45471509e972e2 WatchSource:0}: Error finding container 2d5ce33b9f5e50f324b0ba1d91768db58788a5ad5dedb285fd45471509e972e2: Status 404 returned error can't find the container with id 2d5ce33b9f5e50f324b0ba1d91768db58788a5ad5dedb285fd45471509e972e2
Sep 05 02:18:20.581159 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:20.581146    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf1afb938f794d571614842ff701595.slice/crio-eb5d22a51cf3edc36dcd326fa586913596172d293570544b2bd1cb06ecb13374 WatchSource:0}: Error finding container eb5d22a51cf3edc36dcd326fa586913596172d293570544b2bd1cb06ecb13374: Status 404 returned error can't find the container with id eb5d22a51cf3edc36dcd326fa586913596172d293570544b2bd1cb06ecb13374
Sep 05 02:18:20.585314 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.585293    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:18:20.592924 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.592907    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.632780 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.632725    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal" event={"ID":"63872a04a76bb538711c93178fe3d471","Type":"ContainerStarted","Data":"2d5ce33b9f5e50f324b0ba1d91768db58788a5ad5dedb285fd45471509e972e2"}
Sep 05 02:18:20.633822 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.633805    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" event={"ID":"5bf1afb938f794d571614842ff701595","Type":"ContainerStarted","Data":"eb5d22a51cf3edc36dcd326fa586913596172d293570544b2bd1cb06ecb13374"}
Sep 05 02:18:20.652576 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:20.652548    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:20.693757 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.693734    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.794190 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.794169    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.894730 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.894677    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:20.995466 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:20.995422    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-68.ec2.internal\" not found"
Sep 05 02:18:21.048872 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:21.048846    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:21.105805 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:21.105772    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:22.512213 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.512047    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:22.513836 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.513660    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal"
Sep 05 02:18:22.522090 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.522046    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:22.882323 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.882289    2568 apiserver.go:52] "Watching apiserver"
Sep 05 02:18:22.890759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.890735    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:18:22.891160 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.891137    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pvm72","kube-system/konnectivity-agent-pkhz9","kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal","openshift-image-registry/node-ca-5fl8m","openshift-multus/multus-additional-cni-plugins-qsm2d","openshift-multus/multus-h4qn5","openshift-network-operator/iptables-alerter-vq95w","openshift-ovn-kubernetes/ovnkube-node-wlh74","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb","openshift-cluster-node-tuning-operator/tuned-txvd8","openshift-dns/node-resolver-5z4zs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal","openshift-multus/network-metrics-daemon-jr2fh","openshift-network-diagnostics/network-check-target-kd2fl"]
Sep 05 02:18:22.893588 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.893567    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:22.895745 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.895726    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:22.895840 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:22.895793    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:22.897562 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.897540    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6wrt7\""
Sep 05 02:18:22.897682 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.897657    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:18:22.897793 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.897756    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.898125 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.897950    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.898125 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.898088    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:22.898267 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:22.898159    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:22.899956 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.899938    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:22.901799 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.901630    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:22.902903 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.902883    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.904458 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.904440    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6hvn\""
Sep 05 02:18:22.904707 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.904691    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.905088 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.905060    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:22.906143 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.906093    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:18:22.906143 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.906120    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.906361 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.906347    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tmr87\""
Sep 05 02:18:22.906915 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.906893    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:18:22.907149 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.907129    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:18:22.907316 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.907298    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.907482 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.907465    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:22.907975 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.907959    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:18:22.909168 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.909150    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gmbh2\""
Sep 05 02:18:22.909815 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.909792    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:22.909880 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:22.909859    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:22.912000 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.911984    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:22.912176 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912153    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.912256 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.912256 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912220    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdzg\" (UniqueName: \"kubernetes.io/projected/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-kube-api-access-fmdzg\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.912365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912268    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cnibin\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.912365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912296    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-iptables-alerter-script\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:22.912365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:22.912365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912359    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e81178e-5034-4b0b-9009-8a964e5462ed-host\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:22.912548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912401    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.912548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912426    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlrb\" (UniqueName: \"kubernetes.io/projected/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-kube-api-access-stlrb\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:22.912548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7w4\" (UniqueName: \"kubernetes.io/projected/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-kube-api-access-8k7w4\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:22.912548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912487    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e81178e-5034-4b0b-9009-8a964e5462ed-serviceca\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:22.912548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912520    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.912787 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912551    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:22.912787 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrk7\" (UniqueName: \"kubernetes.io/projected/4e81178e-5034-4b0b-9009-8a964e5462ed-kube-api-access-kcrk7\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:22.912787 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912629    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-host-slash\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:22.912787 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.912652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-system-cni-dir\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:22.913702 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.913683    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.914357 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.914316    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.916340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.915301    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:18:22.916340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.915393    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:18:22.916340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.915479    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.916340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.915861    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:22.916340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.916156    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m4tnv\""
Sep 05 02:18:22.916340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.916273    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zqlfn\""
Sep 05 02:18:22.917261 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.917232    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:18:22.917648 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.917631    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:18:22.917743 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.917701    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:22.921874 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.921855    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:22.924136 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.924110    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:22.924228 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.924146    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.925277 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925256    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:18:22.925277 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925269    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.925509 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925491    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.925562 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925522    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.925562 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925495    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.925650 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925590    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:18:22.925650 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925604    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.925714 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925700    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:18:22.925743 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925719    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:18:22.925786 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925770    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7ndj9\""
Sep 05 02:18:22.925824 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925810    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qbb6b\""
Sep 05 02:18:22.926041 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.925911    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-46kfj\""
Sep 05 02:18:22.964238 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.964220    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zm6k8"
Sep 05 02:18:22.975478 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:22.975460    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zm6k8"
Sep 05 02:18:23.002303 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.002283    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:18:23.013727 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013709    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-systemd-units\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.013824 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013736    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:23.013824 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysctl-d\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.013824 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013785    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-run-netns\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013821    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-var-lib-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013842    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-ovn\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013856    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-ovnkube-config\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.013872    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013895    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e81178e-5034-4b0b-9009-8a964e5462ed-serviceca\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013931    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-system-cni-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.013972 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.013955    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:23.513933167 +0000 UTC m=+2.941049386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.013995    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-conf-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014023    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c869803e-5bb5-4227-a6e8-07c97cf8193f-kubelet-config\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014050    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysconfig\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014088    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-systemd\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014127    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-daemon-config\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014161    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c869803e-5bb5-4227-a6e8-07c97cf8193f-dbus\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqsj\" (UniqueName: \"kubernetes.io/projected/02284cc8-068b-431e-a769-3ddd6faff502-kube-api-access-7jqsj\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014218    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-cni-bin\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014280    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-etc-kubernetes\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.014313 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014312    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-tuned\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014338    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-etc-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014364    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-kubernetes\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014384    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-run\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/829859f0-9e5a-4624-bc4c-ab78a0d0d2f3-konnectivity-ca\") pod \"konnectivity-agent-pkhz9\" (UID: \"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3\") " pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014442    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlrb\" (UniqueName: \"kubernetes.io/projected/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-kube-api-access-stlrb\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014468    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7w4\" (UniqueName: \"kubernetes.io/projected/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-kube-api-access-8k7w4\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014497    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e81178e-5034-4b0b-9009-8a964e5462ed-serviceca\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014511    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbg6\" (UniqueName: \"kubernetes.io/projected/93145336-7051-4bac-9cfd-7899e891fc2e-kube-api-access-ntbg6\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014538    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-lib-modules\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014582    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-log-socket\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.014846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.014606    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-cni-netd\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015097    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-sys-fs\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015136    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-modprobe-d\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015159    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-var-lib-kubelet\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015201    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-system-cni-dir\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015231    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-slash\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015278    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cnibin\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015300    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e81178e-5034-4b0b-9009-8a964e5462ed-host\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015322    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-k8s-cni-cncf-io\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015347    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-cni-multus\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.015391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015378    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-kubelet\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015401    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-device-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015430    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a22e26a-0af5-4f77-83a8-4a7f79117968-tmp\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02284cc8-068b-431e-a769-3ddd6faff502-tmp-dir\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015501    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015527    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-host\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-run-ovn-kubernetes\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015576    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-ovnkube-script-lib\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015601    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrk7\" (UniqueName: \"kubernetes.io/projected/4e81178e-5034-4b0b-9009-8a964e5462ed-kube-api-access-kcrk7\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015625    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-multus-certs\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj69k\" (UniqueName: \"kubernetes.io/projected/084f5a48-1cbe-4328-806e-c96f3150ac7b-kube-api-access-pj69k\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015672    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-socket-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-node-log\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015718    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-env-overrides\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015744    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdzg\" (UniqueName: \"kubernetes.io/projected/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-kube-api-access-fmdzg\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015767    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/084f5a48-1cbe-4328-806e-c96f3150ac7b-cni-binary-copy\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.015937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015789    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysctl-conf\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015814    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-iptables-alerter-script\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015838    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015861    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-socket-dir-parent\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015889    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-sys\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015912    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-kubelet\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015941    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/304abbfc-92b7-4c96-88d3-2a36976c35de-ovn-node-metrics-cert\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015964    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9f4l\" (UniqueName: \"kubernetes.io/projected/304abbfc-92b7-4c96-88d3-2a36976c35de-kube-api-access-l9f4l\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.015987    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-hostroot\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016013    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-registration-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016036    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/829859f0-9e5a-4624-bc4c-ab78a0d0d2f3-agent-certs\") pod \"konnectivity-agent-pkhz9\" (UID: \"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3\") " pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016058    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-systemd\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016140    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-cni-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016168    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016193    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02284cc8-068b-431e-a769-3ddd6faff502-hosts-file\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.016730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016216    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-cni-bin\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016240    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-host-slash\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016298    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016324    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016351    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-cnibin\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016373    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-netns\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016396    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016420    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vbq\" (UniqueName: \"kubernetes.io/projected/7a22e26a-0af5-4f77-83a8-4a7f79117968-kube-api-access-p4vbq\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016538    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-system-cni-dir\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016603    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cnibin\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016646    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e81178e-5034-4b0b-9009-8a964e5462ed-host\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.017479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.016842    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.018049 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.017876    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-iptables-alerter-script\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.018316 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.018289    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-host-slash\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.018426 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.018387    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.018681 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.018660    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.018740 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.018726    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.022700 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.022678    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:18:23.026742 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.026721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlrb\" (UniqueName: \"kubernetes.io/projected/4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d-kube-api-access-stlrb\") pod \"iptables-alerter-vq95w\" (UID: \"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d\") " pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.026950 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.026924    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7w4\" (UniqueName: \"kubernetes.io/projected/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-kube-api-access-8k7w4\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:23.028066 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.028048    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:23.028066 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.028085    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:23.028222 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.028100    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:23.033091 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.028159    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:23.52814129 +0000 UTC m=+2.955257496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:23.034217 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.034190    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrk7\" (UniqueName: \"kubernetes.io/projected/4e81178e-5034-4b0b-9009-8a964e5462ed-kube-api-access-kcrk7\") pod \"node-ca-5fl8m\" (UID: \"4e81178e-5034-4b0b-9009-8a964e5462ed\") " pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.035356 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.035335    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdzg\" (UniqueName: \"kubernetes.io/projected/085eba4e-9a38-49ae-a3a9-e7aab3aee16b-kube-api-access-fmdzg\") pod \"multus-additional-cni-plugins-qsm2d\" (UID: \"085eba4e-9a38-49ae-a3a9-e7aab3aee16b\") " pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.116659 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116627    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-k8s-cni-cncf-io\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.116659 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116663    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-cni-multus\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116678    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-kubelet\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116698    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-device-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116725    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a22e26a-0af5-4f77-83a8-4a7f79117968-tmp\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116748    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02284cc8-068b-431e-a769-3ddd6faff502-tmp-dir\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116748    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-cni-multus\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116764    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-kubelet\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116770    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-host\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116821    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-device-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-host\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116755    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-k8s-cni-cncf-io\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.116863 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116860    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-run-ovn-kubernetes\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116891    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-ovnkube-script-lib\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116918    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-run-ovn-kubernetes\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116919    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-multus-certs\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116959    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-multus-certs\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.116961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj69k\" (UniqueName: \"kubernetes.io/projected/084f5a48-1cbe-4328-806e-c96f3150ac7b-kube-api-access-pj69k\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117002    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-socket-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117042    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02284cc8-068b-431e-a769-3ddd6faff502-tmp-dir\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-node-log\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117112    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-env-overrides\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-node-log\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117137    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/084f5a48-1cbe-4328-806e-c96f3150ac7b-cni-binary-copy\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117145    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-socket-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysctl-conf\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117208    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-socket-dir-parent\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117231    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-sys\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117262    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-kubelet\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117281    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysctl-conf\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.117410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/304abbfc-92b7-4c96-88d3-2a36976c35de-ovn-node-metrics-cert\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9f4l\" (UniqueName: \"kubernetes.io/projected/304abbfc-92b7-4c96-88d3-2a36976c35de-kube-api-access-l9f4l\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-hostroot\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117402    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-registration-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/829859f0-9e5a-4624-bc4c-ab78a0d0d2f3-agent-certs\") pod \"konnectivity-agent-pkhz9\" (UID: \"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3\") " pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-systemd\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117492    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-cni-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117523    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-env-overrides\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117534    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117543    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-socket-dir-parent\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117560    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02284cc8-068b-431e-a769-3ddd6faff502-hosts-file\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117579    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-systemd\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117584    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-cni-bin\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117611    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-kubelet\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117453    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-ovnkube-script-lib\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.118292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-cnibin\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117691    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-cnibin\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117699    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-netns\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117725    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117724    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/084f5a48-1cbe-4328-806e-c96f3150ac7b-cni-binary-copy\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117732    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-hostroot\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.117744    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117749    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vbq\" (UniqueName: \"kubernetes.io/projected/7a22e26a-0af5-4f77-83a8-4a7f79117968-kube-api-access-p4vbq\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117774    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-registration-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-systemd-units\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.117805    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:23.617788535 +0000 UTC m=+3.044904761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117812    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-sys\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117833    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02284cc8-068b-431e-a769-3ddd6faff502-hosts-file\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117849    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-systemd-units\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117879    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-cni-bin\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117656    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-cni-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117913    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysctl-d\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-run-netns\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117893    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-run-netns\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-var-lib-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117980    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.117985    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-ovn\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118015    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-ovn\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-ovnkube-config\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118049    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-run-netns\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118066    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-system-cni-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118093    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysctl-d\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-conf-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118110    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-var-lib-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118149    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c869803e-5bb5-4227-a6e8-07c97cf8193f-kubelet-config\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118157    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-system-cni-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysconfig\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118174    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-conf-dir\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118200    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-systemd\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118210    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c869803e-5bb5-4227-a6e8-07c97cf8193f-kubelet-config\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118218    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-sysconfig\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.119781 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118228    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-daemon-config\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118251    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c869803e-5bb5-4227-a6e8-07c97cf8193f-dbus\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118269    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-systemd\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqsj\" (UniqueName: \"kubernetes.io/projected/02284cc8-068b-431e-a769-3ddd6faff502-kube-api-access-7jqsj\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118345    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-cni-bin\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118389    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c869803e-5bb5-4227-a6e8-07c97cf8193f-dbus\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118407    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-etc-kubernetes\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118422    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-run-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118432    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-tuned\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118456    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-host-var-lib-cni-bin\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-etc-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118496    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-kubernetes\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118500    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-etc-openvswitch\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-run\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/829859f0-9e5a-4624-bc4c-ab78a0d0d2f3-konnectivity-ca\") pod \"konnectivity-agent-pkhz9\" (UID: \"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3\") " pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118542    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/304abbfc-92b7-4c96-88d3-2a36976c35de-ovnkube-config\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118549    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/084f5a48-1cbe-4328-806e-c96f3150ac7b-etc-kubernetes\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.120491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118549    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbg6\" (UniqueName: \"kubernetes.io/projected/93145336-7051-4bac-9cfd-7899e891fc2e-kube-api-access-ntbg6\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118595    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-lib-modules\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118615    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-log-socket\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-cni-netd\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-sys-fs\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-modprobe-d\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118697    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-log-socket\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118702    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-var-lib-kubelet\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118736    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118746    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-var-lib-kubelet\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-slash\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118833    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-modprobe-d\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118843    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-slash\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118877    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-run\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118883    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/304abbfc-92b7-4c96-88d3-2a36976c35de-host-cni-netd\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118931    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-lib-modules\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118944    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-kubernetes\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.118958    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93145336-7051-4bac-9cfd-7899e891fc2e-sys-fs\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.121846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.119146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/084f5a48-1cbe-4328-806e-c96f3150ac7b-multus-daemon-config\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.121846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.119327    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/829859f0-9e5a-4624-bc4c-ab78a0d0d2f3-konnectivity-ca\") pod \"konnectivity-agent-pkhz9\" (UID: \"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3\") " pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.121846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.119874    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/304abbfc-92b7-4c96-88d3-2a36976c35de-ovn-node-metrics-cert\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.121846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.120164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a22e26a-0af5-4f77-83a8-4a7f79117968-tmp\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.121846 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.120950    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a22e26a-0af5-4f77-83a8-4a7f79117968-etc-tuned\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.122130 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.122113    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/829859f0-9e5a-4624-bc4c-ab78a0d0d2f3-agent-certs\") pod \"konnectivity-agent-pkhz9\" (UID: \"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3\") " pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.126119 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.126054    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj69k\" (UniqueName: \"kubernetes.io/projected/084f5a48-1cbe-4328-806e-c96f3150ac7b-kube-api-access-pj69k\") pod \"multus-h4qn5\" (UID: \"084f5a48-1cbe-4328-806e-c96f3150ac7b\") " pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.130208 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.130174    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9f4l\" (UniqueName: \"kubernetes.io/projected/304abbfc-92b7-4c96-88d3-2a36976c35de-kube-api-access-l9f4l\") pod \"ovnkube-node-wlh74\" (UID: \"304abbfc-92b7-4c96-88d3-2a36976c35de\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.130277 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.130252    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbg6\" (UniqueName: \"kubernetes.io/projected/93145336-7051-4bac-9cfd-7899e891fc2e-kube-api-access-ntbg6\") pod \"aws-ebs-csi-driver-node-l6rmb\" (UID: \"93145336-7051-4bac-9cfd-7899e891fc2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.130493 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.130466    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqsj\" (UniqueName: \"kubernetes.io/projected/02284cc8-068b-431e-a769-3ddd6faff502-kube-api-access-7jqsj\") pod \"node-resolver-5z4zs\" (UID: \"02284cc8-068b-431e-a769-3ddd6faff502\") " pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.130656 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.130642    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vbq\" (UniqueName: \"kubernetes.io/projected/7a22e26a-0af5-4f77-83a8-4a7f79117968-kube-api-access-p4vbq\") pod \"tuned-txvd8\" (UID: \"7a22e26a-0af5-4f77-83a8-4a7f79117968\") " pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.205379 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.205310    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vq95w"
Sep 05 02:18:23.214840 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.214819    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5fl8m"
Sep 05 02:18:23.225725 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.225702    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qsm2d"
Sep 05 02:18:23.230480 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.230452    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4qn5"
Sep 05 02:18:23.236179 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.236159    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb"
Sep 05 02:18:23.243867 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.243849    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:23.251290 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.251270    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-txvd8"
Sep 05 02:18:23.257793 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.257772    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5z4zs"
Sep 05 02:18:23.263372 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.263350    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:23.271435 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.271417    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:23.521925 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.521842    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:23.522085 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.521951    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:23.522085 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.522000    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:24.521987452 +0000 UTC m=+3.949103659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:23.622716 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.622685    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:23.622881 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.622735    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:23.622881 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.622865    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:23.622881 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.622881    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:23.623043 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.622886    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:23.623043 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.622890    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:23.623043 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.622944    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:24.622927839 +0000 UTC m=+4.050044055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:23.623043 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:23.622959    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:24.622953032 +0000 UTC m=+4.050069238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:23.814950 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:23.814828    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a22e26a_0af5_4f77_83a8_4a7f79117968.slice/crio-6a26177a934d4422e9fdc2abf92d0afeda0e45139266b51b4264a5154909c74d WatchSource:0}: Error finding container 6a26177a934d4422e9fdc2abf92d0afeda0e45139266b51b4264a5154909c74d: Status 404 returned error can't find the container with id 6a26177a934d4422e9fdc2abf92d0afeda0e45139266b51b4264a5154909c74d
Sep 05 02:18:23.816834 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:23.816813    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829859f0_9e5a_4624_bc4c_ab78a0d0d2f3.slice/crio-860d1f5c7de5ac657ac173958ec5e9190cf23e7900850dd0fdee50af2047ff51 WatchSource:0}: Error finding container 860d1f5c7de5ac657ac173958ec5e9190cf23e7900850dd0fdee50af2047ff51: Status 404 returned error can't find the container with id 860d1f5c7de5ac657ac173958ec5e9190cf23e7900850dd0fdee50af2047ff51
Sep 05 02:18:23.817572 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:23.817541    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93145336_7051_4bac_9cfd_7899e891fc2e.slice/crio-f6256b4dad816ec708ce63705a4f4957dde32e29d97d16bcf24c92cf2d1f46ce WatchSource:0}: Error finding container f6256b4dad816ec708ce63705a4f4957dde32e29d97d16bcf24c92cf2d1f46ce: Status 404 returned error can't find the container with id f6256b4dad816ec708ce63705a4f4957dde32e29d97d16bcf24c92cf2d1f46ce
Sep 05 02:18:23.818426 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:23.818059    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084f5a48_1cbe_4328_806e_c96f3150ac7b.slice/crio-ac283250842fd83fa0c318375247b578d8447a643d7143a1f211ba48425b912c WatchSource:0}: Error finding container ac283250842fd83fa0c318375247b578d8447a643d7143a1f211ba48425b912c: Status 404 returned error can't find the container with id ac283250842fd83fa0c318375247b578d8447a643d7143a1f211ba48425b912c
Sep 05 02:18:23.978966 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.978919    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:22 +0000 UTC" deadline="2028-04-29 14:07:37.067524174 +0000 UTC"
Sep 05 02:18:23.978966 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:23.978964    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14459h49m13.088563109s"
Sep 05 02:18:24.037739 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.037709    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vq95w" event={"ID":"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d","Type":"ContainerStarted","Data":"8db86795b5aed589d4fa8ed4c8bd874619213a226280d5cc5529e0a56136a3cc"}
Sep 05 02:18:24.039435 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.039405    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"5f7479b4f9112f8c6aab924b51b9ff804ecd3e19fc6bc7eec5234ca278374af8"}
Sep 05 02:18:24.040537 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.040507    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5z4zs" event={"ID":"02284cc8-068b-431e-a769-3ddd6faff502","Type":"ContainerStarted","Data":"dd9feb318ec1b71ea84b3ac23ab7824d33623c6716b63ad15e14dbce7aed983f"}
Sep 05 02:18:24.041628 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.041604    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5fl8m" event={"ID":"4e81178e-5034-4b0b-9009-8a964e5462ed","Type":"ContainerStarted","Data":"f330d2d3a2c19c803811ec359ada4aa3de2c71a5c4480ffb77606ecfa7f38023"}
Sep 05 02:18:24.042685 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.042661    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pkhz9" event={"ID":"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3","Type":"ContainerStarted","Data":"860d1f5c7de5ac657ac173958ec5e9190cf23e7900850dd0fdee50af2047ff51"}
Sep 05 02:18:24.043814 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.043786    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-txvd8" event={"ID":"7a22e26a-0af5-4f77-83a8-4a7f79117968","Type":"ContainerStarted","Data":"6a26177a934d4422e9fdc2abf92d0afeda0e45139266b51b4264a5154909c74d"}
Sep 05 02:18:24.044916 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.044891    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerStarted","Data":"50ac73b52a992e3a2a8ead45155a407416d94737771e89ad44134fe62c19f3ad"}
Sep 05 02:18:24.045968 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.045949    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4qn5" event={"ID":"084f5a48-1cbe-4328-806e-c96f3150ac7b","Type":"ContainerStarted","Data":"ac283250842fd83fa0c318375247b578d8447a643d7143a1f211ba48425b912c"}
Sep 05 02:18:24.047050 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.047029    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb" event={"ID":"93145336-7051-4bac-9cfd-7899e891fc2e","Type":"ContainerStarted","Data":"f6256b4dad816ec708ce63705a4f4957dde32e29d97d16bcf24c92cf2d1f46ce"}
Sep 05 02:18:24.049025 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.049000    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal" event={"ID":"63872a04a76bb538711c93178fe3d471","Type":"ContainerStarted","Data":"2f31d5e7640b9751395d0f3bb074f4d301b5435801a1befd5567ad78961c0a4e"}
Sep 05 02:18:24.063742 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.063682    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.ec2.internal" podStartSLOduration=2.063671016 podStartE2EDuration="2.063671016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:24.063432746 +0000 UTC m=+3.490548971" watchObservedRunningTime="2026-09-05 02:18:24.063671016 +0000 UTC m=+3.490787240"
Sep 05 02:18:24.529682 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.529619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:24.529874 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.529787    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:24.529874 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.529849    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.529830386 +0000 UTC m=+5.956946590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:24.630905 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.630866    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:24.631121 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.630919    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:24.631121 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.631104    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:24.631252 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.631169    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.631148517 +0000 UTC m=+6.058264721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:24.631252 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.631218    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:24.631365 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.631270    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:24.631365 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.631307    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:24.631365 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:24.631361    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.63134535 +0000 UTC m=+6.058461567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:24.979295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.979219    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:22 +0000 UTC" deadline="2028-03-05 21:01:25.028567841 +0000 UTC"
Sep 05 02:18:24.979295 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:24.979264    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13146h43m0.049308007s"
Sep 05 02:18:25.028748 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:25.027968    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:25.028748 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:25.028109    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:25.028748 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:25.028201    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:25.028748 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:25.028302    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:25.028748 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:25.028355    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:25.028748 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:25.028421    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:25.072954 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:25.072920    2568 generic.go:356] "Generic (PLEG): container finished" podID="5bf1afb938f794d571614842ff701595" containerID="1762d033e0b559bad602ef5da4f4425560d8a56585127b892f054615dc3c9ca4" exitCode=0
Sep 05 02:18:25.073908 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:25.073878    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" event={"ID":"5bf1afb938f794d571614842ff701595","Type":"ContainerDied","Data":"1762d033e0b559bad602ef5da4f4425560d8a56585127b892f054615dc3c9ca4"}
Sep 05 02:18:26.096849 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:26.096617    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" event={"ID":"5bf1afb938f794d571614842ff701595","Type":"ContainerStarted","Data":"35a9afc76fc83a7d752c7979a5dd39e6b9a44cda8a4988050bcbb7ca805d0208"}
Sep 05 02:18:26.548204 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:26.547653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:26.548204 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.547804    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:26.548204 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.547855    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:30.547842883 +0000 UTC m=+9.974959086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:26.648177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:26.648224    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.648400    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.648461    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:30.648443804 +0000 UTC m=+10.075560021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.648850    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.648870    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.648884    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:26.649018 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:26.648928    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:30.64891181 +0000 UTC m=+10.076028018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:27.026821 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:27.026787    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:27.027000 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:27.026923    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:27.027365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:27.027335    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:27.027488 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:27.027426    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:27.027572 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:27.027556    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:27.027661 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:27.027641    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:29.025204 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:29.025166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:29.025657 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:29.025314    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:29.025657 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:29.025369    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:29.025657 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:29.025420    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:29.025657 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:29.025513    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:29.025657 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:29.025594    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:30.584880 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:30.584845    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:30.585386 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.584963    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:30.585386 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.585033    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:38.585013338 +0000 UTC m=+18.012129546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:30.685958    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:30.686011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.686187    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.686202    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.686220    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.686232    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.686256    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:38.68623767 +0000 UTC m=+18.113353893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:30.686572 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:30.686271    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:38.686261024 +0000 UTC m=+18.113377243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:31.026213 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:31.026160    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:31.026213 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:31.026199    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:31.026441 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:31.026159    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:31.026441 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:31.026292    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:31.026441 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:31.026376    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:31.026649 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:31.026452    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:33.025351 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:33.025317    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:33.025731 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:33.025317    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:33.025731 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:33.025434    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:33.025731 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:33.025330    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:33.025731 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:33.025500    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:33.025731 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:33.025579    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:35.024819 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:35.024783    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:35.024819 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:35.024814    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:35.025315 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:35.024784    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:35.025315 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:35.024898    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:35.025315 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:35.024975    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:35.025315 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:35.025037    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:37.025207 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:37.025166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:37.025657 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:37.025251    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:37.025657 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:37.025239    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:37.025657 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:37.025373    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:37.025657 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:37.025425    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:37.025657 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:37.025507    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:38.647673 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:38.647632    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:38.648162 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.647762    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:38.648162 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.647823    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.647807046 +0000 UTC m=+34.074923270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:38.748961 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:38.748924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:38.749156 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:38.748971    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:38.749156 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.749098    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:38.749156 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.749124    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:38.749156 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.749130    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:38.749156 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.749140    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:38.749357 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.749197    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.749177738 +0000 UTC m=+34.176293957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:38.749357 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:38.749215    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.749205914 +0000 UTC m=+34.176322123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:39.025595 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:39.025509    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:39.025595 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:39.025545    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:39.025811 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:39.025509    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:39.025811 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:39.025649    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:39.025811 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:39.025730    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:39.025941 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:39.025813    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:41.025864 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:41.025826    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:41.026286 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:41.025945    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:41.026348 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:41.026334    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:41.026441 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:41.026425    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:41.026487 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:41.026472    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:41.026554 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:41.026536    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:42.127847 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.127758    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerStarted","Data":"ae381a214f77974a824abde94fd918cfd9db3b1d8dd57085ec6a3d4efdd21ad1"}
Sep 05 02:18:42.129289 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.129231    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb" event={"ID":"93145336-7051-4bac-9cfd-7899e891fc2e","Type":"ContainerStarted","Data":"749882f54b3e2aa2b455b2eba8e34d8c057b29efd220a276d7a15ff507134684"}
Sep 05 02:18:42.130701 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.130636    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"a08c5598ee85b36cdf8763d64cfd56adbb649e12668f090792aeeab62db0875b"}
Sep 05 02:18:42.134099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.133820    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5z4zs" event={"ID":"02284cc8-068b-431e-a769-3ddd6faff502","Type":"ContainerStarted","Data":"49df2c89f803710c8db1c47af9ee4581b03a97a07d4ce1a19f79a4c4770350ed"}
Sep 05 02:18:42.136014 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.135992    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5fl8m" event={"ID":"4e81178e-5034-4b0b-9009-8a964e5462ed","Type":"ContainerStarted","Data":"d21f54ec327a61ae41fe64b341d28a1239b0fe64edb37f1416a7f35c366e6eac"}
Sep 05 02:18:42.137942 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.137918    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pkhz9" event={"ID":"829859f0-9e5a-4624-bc4c-ab78a0d0d2f3","Type":"ContainerStarted","Data":"ac8c5c3e8b879ced4f9600e2fed434ead55f893fbfc27f1263e2c8e0806f1b85"}
Sep 05 02:18:42.139548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.139522    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-txvd8" event={"ID":"7a22e26a-0af5-4f77-83a8-4a7f79117968","Type":"ContainerStarted","Data":"287a12df50bc420ff12105ee5d3aa52c51f38b69dc46ac666b810e1a85f03333"}
Sep 05 02:18:42.150201 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.150150    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.ec2.internal" podStartSLOduration=20.150138072 podStartE2EDuration="20.150138072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:26.114751155 +0000 UTC m=+5.541867378" watchObservedRunningTime="2026-09-05 02:18:42.150138072 +0000 UTC m=+21.577254298"
Sep 05 02:18:42.164035 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.163987    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5fl8m" podStartSLOduration=3.179180689 podStartE2EDuration="21.163970699s" totalImagesPullingTime="17.98479001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:42.163340177 +0000 UTC m=+21.590456403" watchObservedRunningTime="2026-09-05 02:18:42.163970699 +0000 UTC m=+21.591086924"
Sep 05 02:18:42.180133 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.180040    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-txvd8" podStartSLOduration=3.16977941 podStartE2EDuration="21.180020909s" totalImagesPullingTime="18.010241499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:42.179361782 +0000 UTC m=+21.606478035" watchObservedRunningTime="2026-09-05 02:18:42.180020909 +0000 UTC m=+21.607137136"
Sep 05 02:18:42.198907 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.196997    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pkhz9" podStartSLOduration=10.777387859 podStartE2EDuration="21.196976446s" totalImagesPullingTime="10.419588587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:42.196267982 +0000 UTC m=+21.623384208" watchObservedRunningTime="2026-09-05 02:18:42.196976446 +0000 UTC m=+21.624092675"
Sep 05 02:18:42.216425 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.216370    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5z4zs" podStartSLOduration=3.236227373 podStartE2EDuration="21.216349705s" totalImagesPullingTime="17.980122332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:42.216111707 +0000 UTC m=+21.643227931" watchObservedRunningTime="2026-09-05 02:18:42.216349705 +0000 UTC m=+21.643465937"
Sep 05 02:18:42.546900 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.546874    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:42.547460 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:42.547445    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:43.025183 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.025152    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:43.025336 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.025152    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:43.025336 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:43.025292    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:43.025451 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:43.025385    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:43.025451 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.025445    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:43.025546 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:43.025516    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:43.143057 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.143025    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4qn5" event={"ID":"084f5a48-1cbe-4328-806e-c96f3150ac7b","Type":"ContainerStarted","Data":"da5685097bb1334222694fdff3bac5f2b1134cf4b44c12dab751dac292692d5e"}
Sep 05 02:18:43.144513 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.144493    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vq95w" event={"ID":"4944fbb6-e2c2-4f6a-ac10-9e9414e94d0d","Type":"ContainerStarted","Data":"66220b82e7ab4232c747631ef0394531f2eaf62b5661cd2142248d3f669064bf"}
Sep 05 02:18:43.147566 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.147546    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"11e11b6cd5bf9b4fa3825cd722388e09fd8514eadef8facbac12b0635db6fe4a"}
Sep 05 02:18:43.147644 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.147608    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"e700dfceb88ec977d7c44066a97f70422bd52d155e6f7c6905550dd8efea3a89"}
Sep 05 02:18:43.147692 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.147653    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"e3b6fea65d9a9bf4b55387f25a99c430844b08ce8a224c585b2f17b607069716"}
Sep 05 02:18:43.147692 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.147687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"ad35abeec3e2173b4a6f88ebfd8b89efb89383ae4642ff55a522906ac2f30b96"}
Sep 05 02:18:43.147761 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.147714    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"2a294d2ee5b694a57e19f952ba329e8a51acefe86bca7a717939cf218b95e7b1"}
Sep 05 02:18:43.148844 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.148825    2568 generic.go:356] "Generic (PLEG): container finished" podID="085eba4e-9a38-49ae-a3a9-e7aab3aee16b" containerID="ae381a214f77974a824abde94fd918cfd9db3b1d8dd57085ec6a3d4efdd21ad1" exitCode=0
Sep 05 02:18:43.148916 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.148897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerDied","Data":"ae381a214f77974a824abde94fd918cfd9db3b1d8dd57085ec6a3d4efdd21ad1"}
Sep 05 02:18:43.149480 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.149461    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:43.149632 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.149612    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pkhz9"
Sep 05 02:18:43.160476 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.160441    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h4qn5" podStartSLOduration=3.835699031 podStartE2EDuration="22.160430514s" totalImagesPullingTime="18.324731483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:43.160226648 +0000 UTC m=+22.587342872" watchObservedRunningTime="2026-09-05 02:18:43.160430514 +0000 UTC m=+22.587546738"
Sep 05 02:18:43.187318 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.187287    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vq95w" podStartSLOduration=4.206508002 podStartE2EDuration="22.187274875s" totalImagesPullingTime="17.980766873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:43.186949099 +0000 UTC m=+22.614065323" watchObservedRunningTime="2026-09-05 02:18:43.187274875 +0000 UTC m=+22.614391100"
Sep 05 02:18:43.538046 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.537907    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:43.970960 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.970815    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:43.538042113Z","UUID":"a412c777-6bff-407a-aee0-dea9d18723c3","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:43.972439 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.972418    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:43.972439 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:43.972444    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:44.153981 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:44.153941    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb" event={"ID":"93145336-7051-4bac-9cfd-7899e891fc2e","Type":"ContainerStarted","Data":"148d0ffc51b198f8aa72e813c3189fd98d2c85fd115a689e710e7506b3f0ddd3"}
Sep 05 02:18:45.028624 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:45.028596    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:45.028624 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:45.028631    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:45.028854 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:45.028631    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:45.028854 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:45.028714    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:45.028854 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:45.028790    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:45.028854 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:45.028849    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:45.159294 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:45.159260    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb" event={"ID":"93145336-7051-4bac-9cfd-7899e891fc2e","Type":"ContainerStarted","Data":"d4032176155dbb5dda3e7af34fbb729fb2eb5d517910538101e91da5a5898d6f"}
Sep 05 02:18:45.163927 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:45.163895    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"c916600f0fee42966c9c49a524cd90d047a2d79ffdbe1b63f220939e3548e788"}
Sep 05 02:18:45.181003 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:45.180941    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6rmb" podStartSLOduration=3.751709966 podStartE2EDuration="24.18092691s" totalImagesPullingTime="20.429216944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:45.180390263 +0000 UTC m=+24.607506489" watchObservedRunningTime="2026-09-05 02:18:45.18092691 +0000 UTC m=+24.608043138"
Sep 05 02:18:47.028002 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:47.027972    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:47.028416 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:47.028109    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:47.028416 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:47.028210    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:47.028416 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:47.028234    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:47.028416 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:47.028330    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:47.028589 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:47.028424    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:48.173772 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.173581    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" event={"ID":"304abbfc-92b7-4c96-88d3-2a36976c35de","Type":"ContainerStarted","Data":"63d91d4f561c34e52222f60137cd1af954320a590ca51fdb6ce0670e2f77a4b5"}
Sep 05 02:18:48.174339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.173830    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:48.174339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.173886    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:48.174339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.173934    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:48.175962 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.175931    2568 generic.go:356] "Generic (PLEG): container finished" podID="085eba4e-9a38-49ae-a3a9-e7aab3aee16b" containerID="798e1900bcba38d9c13fce8bbdb6548057a75705370886e661264919f221c523" exitCode=0
Sep 05 02:18:48.176132 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.175964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerDied","Data":"798e1900bcba38d9c13fce8bbdb6548057a75705370886e661264919f221c523"}
Sep 05 02:18:48.190389 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.190366    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:48.192211 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.192190    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:18:48.204146 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:48.204110    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74" podStartSLOduration=9.204665448 podStartE2EDuration="27.204096133s" totalImagesPullingTime="17.999430685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:48.203669407 +0000 UTC m=+27.630785632" watchObservedRunningTime="2026-09-05 02:18:48.204096133 +0000 UTC m=+27.631212352"
Sep 05 02:18:49.024907 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.024859    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:49.024907 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.024877    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:49.024907 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.024893    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:49.025119 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:49.024984    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:49.025119 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:49.025095    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:49.025224 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:49.025184    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:49.073473 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.073446    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvm72"]
Sep 05 02:18:49.076242 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.076212    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jr2fh"]
Sep 05 02:18:49.077007 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.076980    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kd2fl"]
Sep 05 02:18:49.180538 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.180512    2568 generic.go:356] "Generic (PLEG): container finished" podID="085eba4e-9a38-49ae-a3a9-e7aab3aee16b" containerID="07cdd15d15df2754911c8d073ac29cab9e45b079099d5ad49eb1d30a1bf2f773" exitCode=0
Sep 05 02:18:49.181104 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.180608    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerDied","Data":"07cdd15d15df2754911c8d073ac29cab9e45b079099d5ad49eb1d30a1bf2f773"}
Sep 05 02:18:49.181104 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.180654    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:49.181104 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.180660    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:49.181104 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:49.180752    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:49.181104 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:49.180828    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:49.181677 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:49.181318    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:49.181677 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:49.181411    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:50.185768 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:50.185599    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerStarted","Data":"f69e69a912a9e7c331a93f089e7d94a5e5f69a57cb19b85351e68638af45a993"}
Sep 05 02:18:51.026451 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:51.026423    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:51.026636 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:51.026622    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:51.026696 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:51.026647    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:51.026756 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:51.026686    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:51.026756 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:51.026728    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:51.026854 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:51.026788    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:51.189626 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:51.189593    2568 generic.go:356] "Generic (PLEG): container finished" podID="085eba4e-9a38-49ae-a3a9-e7aab3aee16b" containerID="f69e69a912a9e7c331a93f089e7d94a5e5f69a57cb19b85351e68638af45a993" exitCode=0
Sep 05 02:18:51.189916 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:51.189638    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerDied","Data":"f69e69a912a9e7c331a93f089e7d94a5e5f69a57cb19b85351e68638af45a993"}
Sep 05 02:18:53.025044 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:53.025011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:53.025514 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:53.025011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:53.025514 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:53.025158    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr2fh" podUID="dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e"
Sep 05 02:18:53.025514 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:53.025011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:53.025514 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:53.025246    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvm72" podUID="c869803e-5bb5-4227-a6e8-07c97cf8193f"
Sep 05 02:18:53.025514 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:53.025295    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd2fl" podUID="0616d409-0327-43ed-a623-0d55b468f451"
Sep 05 02:18:54.664677 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.664641    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:54.665137 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.664802    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:54.665137 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.664879    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:26.664856696 +0000 UTC m=+66.091972911 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:54.765214 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.765144    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:54.765214 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.765187    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:54.765415 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.765319    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:54.765415 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.765341    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:54.765415 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.765364    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:54.765415 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.765377    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret podName:c869803e-5bb5-4227-a6e8-07c97cf8193f nodeName:}" failed. No retries permitted until 2026-09-05 02:19:26.76536316 +0000 UTC m=+66.192479368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret") pod "global-pull-secret-syncer-pvm72" (UID: "c869803e-5bb5-4227-a6e8-07c97cf8193f") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:54.765415 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.765378    2568 projected.go:196] Error preparing data for projected volume kube-api-access-ql5pz for pod openshift-network-diagnostics/network-check-target-kd2fl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:54.765611 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:54.765432    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz podName:0616d409-0327-43ed-a623-0d55b468f451 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:26.765416262 +0000 UTC m=+66.192532471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql5pz" (UniqueName: "kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz") pod "network-check-target-kd2fl" (UID: "0616d409-0327-43ed-a623-0d55b468f451") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:54.812883 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.812851    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-68.ec2.internal" event="NodeReady"
Sep 05 02:18:54.813028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.812993    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:54.867247 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.867220    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bhwx6"]
Sep 05 02:18:54.901986 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.901958    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x5lgj"]
Sep 05 02:18:54.902162 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.902132    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:54.904635 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.904616    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:54.904771 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.904713    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9hcm\""
Sep 05 02:18:54.904959 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.904943    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:54.915934 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.915910    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bhwx6"]
Sep 05 02:18:54.915934 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.915915    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:54.916093 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.915991    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5lgj"]
Sep 05 02:18:54.919115 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.918757    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:54.919115 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.918845    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:54.919115 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.918849    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:54.919115 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:54.918855    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl8b6\""
Sep 05 02:18:55.024884 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.024814    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:18:55.025012 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.024823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:18:55.025012 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.024956    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:18:55.027553 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.027536    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqlb9\""
Sep 05 02:18:55.027669 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.027622    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rhfc\""
Sep 05 02:18:55.027806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.027789    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:55.027986 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.027970    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:55.028131 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.028115    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:55.028209 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.028178    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:55.067242 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.067217    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.067342 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.067249    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3-cert\") pod \"ingress-canary-x5lgj\" (UID: \"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3\") " pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.067342 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.067281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-config-volume\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.067342 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.067313    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxgm5\" (UniqueName: \"kubernetes.io/projected/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-kube-api-access-jxgm5\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.067342 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.067340    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-tmp-dir\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.067509 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.067369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xcw\" (UniqueName: \"kubernetes.io/projected/6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3-kube-api-access-k7xcw\") pod \"ingress-canary-x5lgj\" (UID: \"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3\") " pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.168150 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168122    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxgm5\" (UniqueName: \"kubernetes.io/projected/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-kube-api-access-jxgm5\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.168280 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168159    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-tmp-dir\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.168280 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xcw\" (UniqueName: \"kubernetes.io/projected/6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3-kube-api-access-k7xcw\") pod \"ingress-canary-x5lgj\" (UID: \"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3\") " pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.168280 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168272    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.168433 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168293    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3-cert\") pod \"ingress-canary-x5lgj\" (UID: \"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3\") " pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.168433 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-config-volume\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.168766 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:55.168738    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:55.168881 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168776    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-tmp-dir\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.168881 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:55.168843    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:55.668828802 +0000 UTC m=+35.095945011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:55.168881 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.168856    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-config-volume\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.173305 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.173283    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3-cert\") pod \"ingress-canary-x5lgj\" (UID: \"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3\") " pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.177595 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.177571    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxgm5\" (UniqueName: \"kubernetes.io/projected/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-kube-api-access-jxgm5\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.177984 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.177966    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xcw\" (UniqueName: \"kubernetes.io/projected/6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3-kube-api-access-k7xcw\") pod \"ingress-canary-x5lgj\" (UID: \"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3\") " pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.225865 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.225837    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5lgj"
Sep 05 02:18:55.560720 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.560588    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5lgj"]
Sep 05 02:18:55.563656 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:18:55.563631    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6078c5c9_bb6c_4f11_8ef2_a94064fdd4e3.slice/crio-8e1dfdd7987a8324766545b3a052c88485953798b4ca8ea1cf5c4e90ded85f88 WatchSource:0}: Error finding container 8e1dfdd7987a8324766545b3a052c88485953798b4ca8ea1cf5c4e90ded85f88: Status 404 returned error can't find the container with id 8e1dfdd7987a8324766545b3a052c88485953798b4ca8ea1cf5c4e90ded85f88
Sep 05 02:18:55.672035 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:55.672004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:55.672425 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:55.672120    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:55.672425 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:55.672191    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:56.672177073 +0000 UTC m=+36.099293276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:56.204769 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:56.204736    2568 generic.go:356] "Generic (PLEG): container finished" podID="085eba4e-9a38-49ae-a3a9-e7aab3aee16b" containerID="e256b572a396e8659910f46735ac68ce77a870b7ad7546a48aa05d458ee94e4f" exitCode=0
Sep 05 02:18:56.204931 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:56.204825    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerDied","Data":"e256b572a396e8659910f46735ac68ce77a870b7ad7546a48aa05d458ee94e4f"}
Sep 05 02:18:56.206539 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:56.206512    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5lgj" event={"ID":"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3","Type":"ContainerStarted","Data":"8e1dfdd7987a8324766545b3a052c88485953798b4ca8ea1cf5c4e90ded85f88"}
Sep 05 02:18:56.682903 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:56.682859    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:56.683473 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:56.682938    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:56.683473 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:56.683025    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:58.683003866 +0000 UTC m=+38.110120084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:57.212596 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:57.212559    2568 generic.go:356] "Generic (PLEG): container finished" podID="085eba4e-9a38-49ae-a3a9-e7aab3aee16b" containerID="39d228bd1629d58749b01a542cb0947e2f71fad7ba04ef6007ec4e3c0f48cbe3" exitCode=0
Sep 05 02:18:57.212789 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:57.212606    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerDied","Data":"39d228bd1629d58749b01a542cb0947e2f71fad7ba04ef6007ec4e3c0f48cbe3"}
Sep 05 02:18:58.218957 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:58.218891    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" event={"ID":"085eba4e-9a38-49ae-a3a9-e7aab3aee16b","Type":"ContainerStarted","Data":"66b42c3bdf6590a21ca967578064fe6d2779f439ca94371251710d0262aec8fd"}
Sep 05 02:18:58.220573 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:58.220550    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5lgj" event={"ID":"6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3","Type":"ContainerStarted","Data":"b460cc77f4d224a1a5dafd9e0458cb7c3586596639af55a810c8297fcda40a63"}
Sep 05 02:18:58.242339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:58.242288    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qsm2d" podStartSLOduration=9.195221393 podStartE2EDuration="37.242274481s" totalImagesPullingTime="28.047053088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:58.241111126 +0000 UTC m=+37.668227349" watchObservedRunningTime="2026-09-05 02:18:58.242274481 +0000 UTC m=+37.669390705"
Sep 05 02:18:58.259154 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:58.259110    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x5lgj" podStartSLOduration=1.8129298820000002 podStartE2EDuration="4.259092325s" totalImagesPullingTime="2.446162443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:58.259034804 +0000 UTC m=+37.686151029" watchObservedRunningTime="2026-09-05 02:18:58.259092325 +0000 UTC m=+37.686208551"
Sep 05 02:18:58.700707 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:18:58.700674    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:18:58.700829 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:58.700791    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:58.700867 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:18:58.700839    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:19:02.700825842 +0000 UTC m=+42.127942045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:02.728008 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:02.727793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:19:02.728380 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:02.727945    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:02.728380 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:02.728101    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:19:10.728065821 +0000 UTC m=+50.155182044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:10.778989 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:10.778951    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:19:10.779453 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:10.779087    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:10.779453 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:10.779153    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:19:26.779138289 +0000 UTC m=+66.206254492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:20.196598 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:20.196571    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlh74"
Sep 05 02:19:26.680323 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.680270    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:19:26.683142 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.683121    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:19:26.691335 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:26.691309    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:19:26.691401 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:26.691379    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs podName:dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e nodeName:}" failed. No retries permitted until 2026-09-05 02:20:30.691362214 +0000 UTC m=+130.118478430 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs") pod "network-metrics-daemon-jr2fh" (UID: "dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e") : secret "metrics-daemon-secret" not found
Sep 05 02:19:26.781340 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.781310    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:19:26.781453 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.781362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:19:26.781453 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:26.781445    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:26.781562 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:26.781488    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.781475933 +0000 UTC m=+98.208592140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:26.781562 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.781486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:19:26.784066 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.784051    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:19:26.784153 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.784136    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:19:26.794513 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.794497    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:19:26.795378 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.795360    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c869803e-5bb5-4227-a6e8-07c97cf8193f-original-pull-secret\") pod \"global-pull-secret-syncer-pvm72\" (UID: \"c869803e-5bb5-4227-a6e8-07c97cf8193f\") " pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:19:26.804539 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.804520    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5pz\" (UniqueName: \"kubernetes.io/projected/0616d409-0327-43ed-a623-0d55b468f451-kube-api-access-ql5pz\") pod \"network-check-target-kd2fl\" (UID: \"0616d409-0327-43ed-a623-0d55b468f451\") " pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:19:26.838939 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.838920    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rhfc\""
Sep 05 02:19:26.847175 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.847155    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:19:26.848388 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:26.848372    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvm72"
Sep 05 02:19:27.123952 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:27.123917    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kd2fl"]
Sep 05 02:19:27.127458 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:19:27.127437    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0616d409_0327_43ed_a623_0d55b468f451.slice/crio-491a7a4c6afbdd097c2504e5f54ab0a2ff4a41ad75b31afed0a3d9de2a52e275 WatchSource:0}: Error finding container 491a7a4c6afbdd097c2504e5f54ab0a2ff4a41ad75b31afed0a3d9de2a52e275: Status 404 returned error can't find the container with id 491a7a4c6afbdd097c2504e5f54ab0a2ff4a41ad75b31afed0a3d9de2a52e275
Sep 05 02:19:27.129096 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:27.129040    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvm72"]
Sep 05 02:19:27.132192 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:19:27.132175    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc869803e_5bb5_4227_a6e8_07c97cf8193f.slice/crio-ecb927de58d7771e75cf1ea7b085b69d2ed0d2bbdeebd7a8996a22966095bec8 WatchSource:0}: Error finding container ecb927de58d7771e75cf1ea7b085b69d2ed0d2bbdeebd7a8996a22966095bec8: Status 404 returned error can't find the container with id ecb927de58d7771e75cf1ea7b085b69d2ed0d2bbdeebd7a8996a22966095bec8
Sep 05 02:19:27.289348 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:27.289311    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kd2fl" event={"ID":"0616d409-0327-43ed-a623-0d55b468f451","Type":"ContainerStarted","Data":"491a7a4c6afbdd097c2504e5f54ab0a2ff4a41ad75b31afed0a3d9de2a52e275"}
Sep 05 02:19:27.290342 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:27.290318    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvm72" event={"ID":"c869803e-5bb5-4227-a6e8-07c97cf8193f","Type":"ContainerStarted","Data":"ecb927de58d7771e75cf1ea7b085b69d2ed0d2bbdeebd7a8996a22966095bec8"}
Sep 05 02:19:32.306807 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:32.306765    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kd2fl" event={"ID":"0616d409-0327-43ed-a623-0d55b468f451","Type":"ContainerStarted","Data":"521b1e0ada9f2006244e248c52b5517640fa62aa1a003e3b5b635bcf99f84e4a"}
Sep 05 02:19:32.307359 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:32.306830    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:19:32.308369 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:32.308342    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvm72" event={"ID":"c869803e-5bb5-4227-a6e8-07c97cf8193f","Type":"ContainerStarted","Data":"0f12b5391aaa081d25227196ebf5de6f30ed2818428fc9dbc5178bc83edf0503"}
Sep 05 02:19:32.350475 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:32.350434    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kd2fl" podStartSLOduration=67.152843681 podStartE2EDuration="1m11.350418953s" totalImagesPullingTime="4.197575272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:32.329149863 +0000 UTC m=+71.756266090" watchObservedRunningTime="2026-09-05 02:19:32.350418953 +0000 UTC m=+71.777535176"
Sep 05 02:19:56.403156 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.402996    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pvm72" podStartSLOduration=91.206445489 podStartE2EDuration="1m35.402980815s" totalImagesPullingTime="4.196535326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:32.351130008 +0000 UTC m=+71.778246232" watchObservedRunningTime="2026-09-05 02:19:56.402980815 +0000 UTC m=+95.830097047"
Sep 05 02:19:56.403650 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.403460    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"]
Sep 05 02:19:56.407531 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.407513    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.412458 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.412433    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 05 02:19:56.412763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.412490    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 05 02:19:56.412763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.412522    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mz5mw\""
Sep 05 02:19:56.412763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.412536    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.412763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.412433    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.420774 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.420752    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"]
Sep 05 02:19:56.479969 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.479944    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.480127 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.479983    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.480127 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.480003    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zts96\" (UniqueName: \"kubernetes.io/projected/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-kube-api-access-zts96\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.517828 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.517794    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"]
Sep 05 02:19:56.520479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.520462    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"
Sep 05 02:19:56.528405 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.528388    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.528499 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.528418    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.530224 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.530210    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ncxmv\""
Sep 05 02:19:56.540237 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.540218    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"]
Sep 05 02:19:56.581199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.581173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.581300 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.581206    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrn6j\" (UniqueName: \"kubernetes.io/projected/59de7810-3c4a-401f-b226-64f409289097-kube-api-access-wrn6j\") pod \"volume-data-source-validator-695c895bd6-5n7gj\" (UID: \"59de7810-3c4a-401f-b226-64f409289097\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"
Sep 05 02:19:56.581300 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.581232    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.581300 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.581248    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zts96\" (UniqueName: \"kubernetes.io/projected/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-kube-api-access-zts96\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.581416 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:56.581371    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:56.581457 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:56.581448    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls podName:9c8ab5ec-55dd-4ad4-8774-85d5a8893aae nodeName:}" failed. No retries permitted until 2026-09-05 02:19:57.081430382 +0000 UTC m=+96.508546608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7mqqs" (UID: "9c8ab5ec-55dd-4ad4-8774-85d5a8893aae") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:56.581836 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.581818    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.591896 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.591873    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zts96\" (UniqueName: \"kubernetes.io/projected/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-kube-api-access-zts96\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:56.681600 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.681533    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn6j\" (UniqueName: \"kubernetes.io/projected/59de7810-3c4a-401f-b226-64f409289097-kube-api-access-wrn6j\") pod \"volume-data-source-validator-695c895bd6-5n7gj\" (UID: \"59de7810-3c4a-401f-b226-64f409289097\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"
Sep 05 02:19:56.695409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.695388    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn6j\" (UniqueName: \"kubernetes.io/projected/59de7810-3c4a-401f-b226-64f409289097-kube-api-access-wrn6j\") pod \"volume-data-source-validator-695c895bd6-5n7gj\" (UID: \"59de7810-3c4a-401f-b226-64f409289097\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"
Sep 05 02:19:56.699972 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.699954    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"]
Sep 05 02:19:56.702819 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.702806    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.705389 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.705367    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.705641 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.705627    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 05 02:19:56.705689 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.705663    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.705929 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.705911    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m958n\""
Sep 05 02:19:56.705980 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.705953    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 05 02:19:56.715117 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.715096    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"]
Sep 05 02:19:56.782350 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.782324    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xw6\" (UniqueName: \"kubernetes.io/projected/c6f68797-eb63-4188-a1a6-f27185930675-kube-api-access-b4xw6\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.782460 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.782356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f68797-eb63-4188-a1a6-f27185930675-serving-cert\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.782460 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.782377    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f68797-eb63-4188-a1a6-f27185930675-config\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.828308 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.828283    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"
Sep 05 02:19:56.883625 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.883590    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xw6\" (UniqueName: \"kubernetes.io/projected/c6f68797-eb63-4188-a1a6-f27185930675-kube-api-access-b4xw6\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.883739 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.883637    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f68797-eb63-4188-a1a6-f27185930675-serving-cert\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.883739 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.883663    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f68797-eb63-4188-a1a6-f27185930675-config\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.884379 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.884328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f68797-eb63-4188-a1a6-f27185930675-config\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.886416 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.886388    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f68797-eb63-4188-a1a6-f27185930675-serving-cert\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:56.892889 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:56.892870    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xw6\" (UniqueName: \"kubernetes.io/projected/c6f68797-eb63-4188-a1a6-f27185930675-kube-api-access-b4xw6\") pod \"service-ca-operator-859f94c4b-t4t8z\" (UID: \"c6f68797-eb63-4188-a1a6-f27185930675\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:57.011251 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:57.011166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"
Sep 05 02:19:57.084507 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:57.084470    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:57.085165 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:57.084643    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:57.085165 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:57.084720    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls podName:9c8ab5ec-55dd-4ad4-8774-85d5a8893aae nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.084699695 +0000 UTC m=+97.511815901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7mqqs" (UID: "9c8ab5ec-55dd-4ad4-8774-85d5a8893aae") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:57.094731 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:57.094708    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj"]
Sep 05 02:19:57.097583 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:19:57.097561    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59de7810_3c4a_401f_b226_64f409289097.slice/crio-7db35f75b23e88490419198549342c3cc23e7de6c3ada37cc0c59dd6c9aa7c88 WatchSource:0}: Error finding container 7db35f75b23e88490419198549342c3cc23e7de6c3ada37cc0c59dd6c9aa7c88: Status 404 returned error can't find the container with id 7db35f75b23e88490419198549342c3cc23e7de6c3ada37cc0c59dd6c9aa7c88
Sep 05 02:19:57.286228 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:57.286134    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z"]
Sep 05 02:19:57.288595 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:19:57.288576    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f68797_eb63_4188_a1a6_f27185930675.slice/crio-c22be3a1b48de85499cfd2cad0a49e43c9df9fa20ab9252ba7883400aeaee18f WatchSource:0}: Error finding container c22be3a1b48de85499cfd2cad0a49e43c9df9fa20ab9252ba7883400aeaee18f: Status 404 returned error can't find the container with id c22be3a1b48de85499cfd2cad0a49e43c9df9fa20ab9252ba7883400aeaee18f
Sep 05 02:19:57.364483 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:57.364435    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z" event={"ID":"c6f68797-eb63-4188-a1a6-f27185930675","Type":"ContainerStarted","Data":"c22be3a1b48de85499cfd2cad0a49e43c9df9fa20ab9252ba7883400aeaee18f"}
Sep 05 02:19:57.365487 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:57.365461    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj" event={"ID":"59de7810-3c4a-401f-b226-64f409289097","Type":"ContainerStarted","Data":"7db35f75b23e88490419198549342c3cc23e7de6c3ada37cc0c59dd6c9aa7c88"}
Sep 05 02:19:58.091753 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:58.091712    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:19:58.092168 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:58.091899    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:58.092168 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:58.091967    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls podName:9c8ab5ec-55dd-4ad4-8774-85d5a8893aae nodeName:}" failed. No retries permitted until 2026-09-05 02:20:00.091951578 +0000 UTC m=+99.519067781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7mqqs" (UID: "9c8ab5ec-55dd-4ad4-8774-85d5a8893aae") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:58.369311 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:58.369274    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj" event={"ID":"59de7810-3c4a-401f-b226-64f409289097","Type":"ContainerStarted","Data":"f97d2b76bab4dd5d6d7e28ad0d11d859afe979c7a6f7ba2cad5bf40f2cfba8b6"}
Sep 05 02:19:58.386795 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:58.386747    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-5n7gj" podStartSLOduration=1.337232559 podStartE2EDuration="2.386732367s" totalImagesPullingTime="1.049499808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:58.385313747 +0000 UTC m=+97.812429971" watchObservedRunningTime="2026-09-05 02:19:58.386732367 +0000 UTC m=+97.813848592"
Sep 05 02:19:58.797377 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:19:58.797274    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:19:58.797543 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:58.797447    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:58.797543 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:19:58.797527    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls podName:d7b1d500-6b4b-49e5-8f80-9ffde88c716c nodeName:}" failed. No retries permitted until 2026-09-05 02:21:02.797506136 +0000 UTC m=+162.224622354 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls") pod "dns-default-bhwx6" (UID: "d7b1d500-6b4b-49e5-8f80-9ffde88c716c") : secret "dns-default-metrics-tls" not found
Sep 05 02:20:00.106093 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:00.106033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:20:00.106504 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:00.106166    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:20:00.106504 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:00.106219    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls podName:9c8ab5ec-55dd-4ad4-8774-85d5a8893aae nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.106205794 +0000 UTC m=+103.533322001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7mqqs" (UID: "9c8ab5ec-55dd-4ad4-8774-85d5a8893aae") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:20:00.375633 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:00.375591    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z" event={"ID":"c6f68797-eb63-4188-a1a6-f27185930675","Type":"ContainerStarted","Data":"58f1113f81703beb5aa161507f38e6ac398a699f6f3aba3051c07178c3e339c9"}
Sep 05 02:20:00.399813 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:00.399760    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z" podStartSLOduration=1.50070631 podStartE2EDuration="4.399745354s" totalImagesPullingTime="2.899039044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:00.398322507 +0000 UTC m=+99.825438733" watchObservedRunningTime="2026-09-05 02:20:00.399745354 +0000 UTC m=+99.826861579"
Sep 05 02:20:01.055142 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.055106    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"]
Sep 05 02:20:01.058033 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.058017    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"
Sep 05 02:20:01.060494 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.060473    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9lbth\""
Sep 05 02:20:01.060611 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.060476    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 05 02:20:01.061473 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.061457    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 05 02:20:01.067094 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.067054    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"]
Sep 05 02:20:01.114735 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.114706    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfwp\" (UniqueName: \"kubernetes.io/projected/5f336f2e-51ff-4753-96d3-e42edb2b3dbc-kube-api-access-czfwp\") pod \"migrator-fdbb8fdff-kptct\" (UID: \"5f336f2e-51ff-4753-96d3-e42edb2b3dbc\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"
Sep 05 02:20:01.202034 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.201998    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pvxmc"]
Sep 05 02:20:01.204995 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.204977    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.207528 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.207503    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:20:01.208264 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.208216    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:20:01.208404 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.208381    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:20:01.208520 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.208404    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:20:01.208588 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.208550    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mx6qs\""
Sep 05 02:20:01.214989 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.214966    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfwp\" (UniqueName: \"kubernetes.io/projected/5f336f2e-51ff-4753-96d3-e42edb2b3dbc-kube-api-access-czfwp\") pod \"migrator-fdbb8fdff-kptct\" (UID: \"5f336f2e-51ff-4753-96d3-e42edb2b3dbc\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"
Sep 05 02:20:01.219346 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.219326    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pvxmc"]
Sep 05 02:20:01.225660 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.225637    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfwp\" (UniqueName: \"kubernetes.io/projected/5f336f2e-51ff-4753-96d3-e42edb2b3dbc-kube-api-access-czfwp\") pod \"migrator-fdbb8fdff-kptct\" (UID: \"5f336f2e-51ff-4753-96d3-e42edb2b3dbc\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"
Sep 05 02:20:01.316000 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.315908    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.316000 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.315950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94rr\" (UniqueName: \"kubernetes.io/projected/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-api-access-b94rr\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.316253 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.316041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.316253 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.316118    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76bea6b1-457b-4002-9182-c09daa07cbf8-crio-socket\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.316253 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.316150    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bea6b1-457b-4002-9182-c09daa07cbf8-data-volume\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.366167 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.366134    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.416619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94rr\" (UniqueName: \"kubernetes.io/projected/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-api-access-b94rr\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.417131    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.417210    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76bea6b1-457b-4002-9182-c09daa07cbf8-crio-socket\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.417227    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bea6b1-457b-4002-9182-c09daa07cbf8-data-volume\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.417306    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.417401    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:01.917381934 +0000 UTC m=+101.344498147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.417443    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.417565    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.417652 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.417607    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:01.917593895 +0000 UTC m=+101.344710115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.418251 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.417687    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bea6b1-457b-4002-9182-c09daa07cbf8-data-volume\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.418251 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.417772    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76bea6b1-457b-4002-9182-c09daa07cbf8-crio-socket\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.426252 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.426229    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94rr\" (UniqueName: \"kubernetes.io/projected/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-api-access-b94rr\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.641760 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.641729    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct"]
Sep 05 02:20:01.644485 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:01.644457    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f336f2e_51ff_4753_96d3_e42edb2b3dbc.slice/crio-5f10e73b1aa642176ed65bcb5f505bde3fe204bb407d003bebfe8ff7cd1398eb WatchSource:0}: Error finding container 5f10e73b1aa642176ed65bcb5f505bde3fe204bb407d003bebfe8ff7cd1398eb: Status 404 returned error can't find the container with id 5f10e73b1aa642176ed65bcb5f505bde3fe204bb407d003bebfe8ff7cd1398eb
Sep 05 02:20:01.922125 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.922006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.922125 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:01.922100    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:01.922355 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.922179    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.922355 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.922234    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:02.922220742 +0000 UTC m=+102.349336950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.922355 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.922232    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.922355 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:01.922299    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:02.922280994 +0000 UTC m=+102.349397212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:02.382995 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:02.382960    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct" event={"ID":"5f336f2e-51ff-4753-96d3-e42edb2b3dbc","Type":"ContainerStarted","Data":"5f10e73b1aa642176ed65bcb5f505bde3fe204bb407d003bebfe8ff7cd1398eb"}
Sep 05 02:20:02.929919 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:02.929891    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:02.930013 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:02.929953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:02.930055 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:02.930022    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.930112 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:02.930055    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:02.930145 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:02.930113    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.930093126 +0000 UTC m=+104.357209343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.930145 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:02.930132    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.930122633 +0000 UTC m=+104.357238839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:03.312690 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.312623    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kd2fl"
Sep 05 02:20:03.388096 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.388047    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct" event={"ID":"5f336f2e-51ff-4753-96d3-e42edb2b3dbc","Type":"ContainerStarted","Data":"97f4fc6d235fb2b117916656aad20ffba31d58d041ad63403d8071047a9f5e43"}
Sep 05 02:20:03.388436 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.388126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct" event={"ID":"5f336f2e-51ff-4753-96d3-e42edb2b3dbc","Type":"ContainerStarted","Data":"2e75ecc04b61d0b816b74ab363b0160611b71ece47a6f2ed4facffd1d81bb6e7"}
Sep 05 02:20:03.405611 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.405565    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kptct" podStartSLOduration=1.156219707 podStartE2EDuration="2.405548981s" totalImagesPullingTime="1.249329274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:03.404435608 +0000 UTC m=+102.831551849" watchObservedRunningTime="2026-09-05 02:20:03.405548981 +0000 UTC m=+102.832665206"
Sep 05 02:20:03.750089 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.750058    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z4zs_02284cc8-068b-431e-a769-3ddd6faff502/dns-node-resolver/0.log"
Sep 05 02:20:03.865254 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.865216    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6mffv"]
Sep 05 02:20:03.881849 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.881828    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6mffv"]
Sep 05 02:20:03.881964 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.881913    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:03.884565 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.884542    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 05 02:20:03.884671 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.884585    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 05 02:20:03.884671 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.884596    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 05 02:20:03.884671 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.884617    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zr76p\""
Sep 05 02:20:03.885474 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.885455    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 05 02:20:03.938319 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.938287    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7d8f7e4-79b6-4b59-af17-97268a29390a-signing-cabundle\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:03.938463 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.938355    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e7d8f7e4-79b6-4b59-af17-97268a29390a-signing-key\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:03.938463 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:03.938385    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxw8\" (UniqueName: \"kubernetes.io/projected/e7d8f7e4-79b6-4b59-af17-97268a29390a-kube-api-access-bxxw8\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.039019 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.038938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxw8\" (UniqueName: \"kubernetes.io/projected/e7d8f7e4-79b6-4b59-af17-97268a29390a-kube-api-access-bxxw8\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.039019 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.039011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7d8f7e4-79b6-4b59-af17-97268a29390a-signing-cabundle\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.039205 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.039050    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e7d8f7e4-79b6-4b59-af17-97268a29390a-signing-key\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.039633 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.039605    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e7d8f7e4-79b6-4b59-af17-97268a29390a-signing-cabundle\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.041629 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.041600    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e7d8f7e4-79b6-4b59-af17-97268a29390a-signing-key\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.047306 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.047283    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxw8\" (UniqueName: \"kubernetes.io/projected/e7d8f7e4-79b6-4b59-af17-97268a29390a-kube-api-access-bxxw8\") pod \"service-ca-75d99c484-6mffv\" (UID: \"e7d8f7e4-79b6-4b59-af17-97268a29390a\") " pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.139982 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.139939    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:20:04.140150 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:04.140109    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:20:04.140189 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:04.140175    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls podName:9c8ab5ec-55dd-4ad4-8774-85d5a8893aae nodeName:}" failed. No retries permitted until 2026-09-05 02:20:12.140159784 +0000 UTC m=+111.567275987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7mqqs" (UID: "9c8ab5ec-55dd-4ad4-8774-85d5a8893aae") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:20:04.191122 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.191087    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-6mffv"
Sep 05 02:20:04.498927 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.497033    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6mffv"]
Sep 05 02:20:04.502494 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:04.502469    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d8f7e4_79b6_4b59_af17_97268a29390a.slice/crio-5fb21ae113f22649e43355ee3cf5a4726af974b82a8ce7bd5a7ed8909201931d WatchSource:0}: Error finding container 5fb21ae113f22649e43355ee3cf5a4726af974b82a8ce7bd5a7ed8909201931d: Status 404 returned error can't find the container with id 5fb21ae113f22649e43355ee3cf5a4726af974b82a8ce7bd5a7ed8909201931d
Sep 05 02:20:04.746568 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.746542    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5fl8m_4e81178e-5034-4b0b-9009-8a964e5462ed/node-ca/0.log"
Sep 05 02:20:04.946912 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.946868    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:04.947146 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:04.946934    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:04.947146 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:04.947018    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:04.947146 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:04.947031    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:04.947146 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:04.947107    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:08.947092621 +0000 UTC m=+108.374208837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:04.947146 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:04.947121    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:08.947114856 +0000 UTC m=+108.374231059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:05.395605 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:05.395559    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-6mffv" event={"ID":"e7d8f7e4-79b6-4b59-af17-97268a29390a","Type":"ContainerStarted","Data":"5b48ed268c8f57c2b994add9ee9ce3f33b4a400baef00132f838b9fd923c1263"}
Sep 05 02:20:05.395754 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:05.395616    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-6mffv" event={"ID":"e7d8f7e4-79b6-4b59-af17-97268a29390a","Type":"ContainerStarted","Data":"5fb21ae113f22649e43355ee3cf5a4726af974b82a8ce7bd5a7ed8909201931d"}
Sep 05 02:20:05.417109 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:05.417036    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-6mffv" podStartSLOduration=2.4170195039999998 podStartE2EDuration="2.417019504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:05.416411269 +0000 UTC m=+104.843527495" watchObservedRunningTime="2026-09-05 02:20:05.417019504 +0000 UTC m=+104.844135728"
Sep 05 02:20:05.946491 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:05.946451    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5lgj_6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3/serve-healthcheck-canary/0.log"
Sep 05 02:20:06.150238 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:06.150206    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kptct_5f336f2e-51ff-4753-96d3-e42edb2b3dbc/migrator/0.log"
Sep 05 02:20:06.352603 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:06.352518    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kptct_5f336f2e-51ff-4753-96d3-e42edb2b3dbc/graceful-termination/0.log"
Sep 05 02:20:08.978647 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:08.978608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:08.979263 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:08.978675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:08.979263 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:08.978749    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:08.979263 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:08.978767    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:08.979263 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:08.978824    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:16.978808752 +0000 UTC m=+116.405924973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:08.979263 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:08.978838    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm podName:76bea6b1-457b-4002-9182-c09daa07cbf8 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:16.978832249 +0000 UTC m=+116.405948451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvxmc" (UID: "76bea6b1-457b-4002-9182-c09daa07cbf8") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:12.202929 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:12.202892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:20:12.203306 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:12.203044    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:20:12.203306 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:12.203132    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls podName:9c8ab5ec-55dd-4ad4-8774-85d5a8893aae nodeName:}" failed. No retries permitted until 2026-09-05 02:20:28.203115227 +0000 UTC m=+127.630231430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7mqqs" (UID: "9c8ab5ec-55dd-4ad4-8774-85d5a8893aae") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:20:17.034807 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.034766    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:17.035208 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.034829    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:17.035303 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.035285    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76bea6b1-457b-4002-9182-c09daa07cbf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:17.037401 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.037377    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76bea6b1-457b-4002-9182-c09daa07cbf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvxmc\" (UID: \"76bea6b1-457b-4002-9182-c09daa07cbf8\") " pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:17.113606 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.113587    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pvxmc"
Sep 05 02:20:17.387561 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.387530    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pvxmc"]
Sep 05 02:20:17.390608 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:17.390579    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bea6b1_457b_4002_9182_c09daa07cbf8.slice/crio-6d33f1725151d43ec78119155cbf625fceb6cf82dd780b1e337613753eae0d71 WatchSource:0}: Error finding container 6d33f1725151d43ec78119155cbf625fceb6cf82dd780b1e337613753eae0d71: Status 404 returned error can't find the container with id 6d33f1725151d43ec78119155cbf625fceb6cf82dd780b1e337613753eae0d71
Sep 05 02:20:17.435754 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:17.435728    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvxmc" event={"ID":"76bea6b1-457b-4002-9182-c09daa07cbf8","Type":"ContainerStarted","Data":"6d33f1725151d43ec78119155cbf625fceb6cf82dd780b1e337613753eae0d71"}
Sep 05 02:20:18.439970 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:18.439941    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvxmc" event={"ID":"76bea6b1-457b-4002-9182-c09daa07cbf8","Type":"ContainerStarted","Data":"eaac3dc4145dc9f27ac8b9cc1a0a9f7d23e63aa047789b580a051a0e84d9059e"}
Sep 05 02:20:18.440345 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:18.439992    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvxmc" event={"ID":"76bea6b1-457b-4002-9182-c09daa07cbf8","Type":"ContainerStarted","Data":"7453e93b999560065ac1ef0efee5480961222d8a0ef2befdda028ec08903e86d"}
Sep 05 02:20:20.448150 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:20.448112    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvxmc" event={"ID":"76bea6b1-457b-4002-9182-c09daa07cbf8","Type":"ContainerStarted","Data":"5a6aa6ef86b628b9362a4eb2c3e6cbbedb8fe0c075d0256db04a31dea893c9a8"}
Sep 05 02:20:20.468431 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:20.468386    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pvxmc" podStartSLOduration=17.498744252 podStartE2EDuration="19.468369724s" totalImagesPullingTime="1.969625472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:20:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:20.467450584 +0000 UTC m=+119.894566809" watchObservedRunningTime="2026-09-05 02:20:20.468369724 +0000 UTC m=+119.895485949"
Sep 05 02:20:24.363263 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.363230    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"]
Sep 05 02:20:24.366530 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.366509    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.370017 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.369993    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 05 02:20:24.370017 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.370007    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 05 02:20:24.370228 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.370061    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 05 02:20:24.370310 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.370289    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 05 02:20:24.370367 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.370305    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:20:24.370367 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.370333    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:20:24.370486 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.370388    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:20:24.385060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.385031    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"]
Sep 05 02:20:24.490344 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.490309    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-hub\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.490520 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.490348    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.490520 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.490376    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4b9399-e958-4695-9baf-969ae979e437-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.490520 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.490461    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-ca\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.490520 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.490497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.490655 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.490605    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5q92\" (UniqueName: \"kubernetes.io/projected/3a4b9399-e958-4695-9baf-969ae979e437-kube-api-access-j5q92\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.591557 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.591527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q92\" (UniqueName: \"kubernetes.io/projected/3a4b9399-e958-4695-9baf-969ae979e437-kube-api-access-j5q92\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.591736 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.591575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-hub\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.591736 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.591591    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.591736 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.591608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4b9399-e958-4695-9baf-969ae979e437-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.591736 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.591649    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-ca\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.591952 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.591762    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.592692 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.592666    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4b9399-e958-4695-9baf-969ae979e437-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.594279 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.594250    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-hub\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.594279 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.594266    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-ca\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.594407 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.594394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.594481 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.594461    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a4b9399-e958-4695-9baf-969ae979e437-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.601638 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.601615    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q92\" (UniqueName: \"kubernetes.io/projected/3a4b9399-e958-4695-9baf-969ae979e437-kube-api-access-j5q92\") pod \"cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd\" (UID: \"3a4b9399-e958-4695-9baf-969ae979e437\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.685374 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.685348    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"
Sep 05 02:20:24.958672 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:24.958580    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd"]
Sep 05 02:20:24.961357 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:24.961326    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a4b9399_e958_4695_9baf_969ae979e437.slice/crio-8f7e0fd543ead40aa51bdfb09cfae4068138239464151ad23137c245f56999f1 WatchSource:0}: Error finding container 8f7e0fd543ead40aa51bdfb09cfae4068138239464151ad23137c245f56999f1: Status 404 returned error can't find the container with id 8f7e0fd543ead40aa51bdfb09cfae4068138239464151ad23137c245f56999f1
Sep 05 02:20:25.464138 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:25.464104    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd" event={"ID":"3a4b9399-e958-4695-9baf-969ae979e437","Type":"ContainerStarted","Data":"8f7e0fd543ead40aa51bdfb09cfae4068138239464151ad23137c245f56999f1"}
Sep 05 02:20:28.225331 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:28.225284    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:20:28.228399 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:28.228370    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c8ab5ec-55dd-4ad4-8774-85d5a8893aae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7mqqs\" (UID: \"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:20:28.518474 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:28.518390    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mz5mw\""
Sep 05 02:20:28.526654 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:28.526625    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"
Sep 05 02:20:28.847838 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:28.847808    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs"]
Sep 05 02:20:28.851066 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:28.851040    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ab5ec_55dd_4ad4_8774_85d5a8893aae.slice/crio-e1b3895b5f44dbc193c6ed41fec32c503736cf9dba51f32aa8cee842b6f9e9ee WatchSource:0}: Error finding container e1b3895b5f44dbc193c6ed41fec32c503736cf9dba51f32aa8cee842b6f9e9ee: Status 404 returned error can't find the container with id e1b3895b5f44dbc193c6ed41fec32c503736cf9dba51f32aa8cee842b6f9e9ee
Sep 05 02:20:29.480542 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:29.480494    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd" event={"ID":"3a4b9399-e958-4695-9baf-969ae979e437","Type":"ContainerStarted","Data":"39c15665ebe78fb08e89dc547beb21714042ab61f5ab983b7d1f7468b4dec528"}
Sep 05 02:20:29.480984 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:29.480551    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd" event={"ID":"3a4b9399-e958-4695-9baf-969ae979e437","Type":"ContainerStarted","Data":"50eda83bce5e52d7bb4f438e60bfe150e45cb6bb23328f13aa43126b271cad4e"}
Sep 05 02:20:29.480984 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:29.480568    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd" event={"ID":"3a4b9399-e958-4695-9baf-969ae979e437","Type":"ContainerStarted","Data":"8c020dba9c305518882d83c2a55ac6aca174df648c66cd31b07a2b5ec009f3aa"}
Sep 05 02:20:29.481850 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:29.481823    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs" event={"ID":"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae","Type":"ContainerStarted","Data":"e1b3895b5f44dbc193c6ed41fec32c503736cf9dba51f32aa8cee842b6f9e9ee"}
Sep 05 02:20:29.499399 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:29.499348    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f9f7f67cf-cfgtd" podStartSLOduration=1.808359883 podStartE2EDuration="5.49932611s" totalImagesPullingTime="3.690966227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:29.498148801 +0000 UTC m=+128.925265032" watchObservedRunningTime="2026-09-05 02:20:29.49932611 +0000 UTC m=+128.926442385"
Sep 05 02:20:30.743979 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:30.743942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:20:30.746503 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:30.746473    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e-metrics-certs\") pod \"network-metrics-daemon-jr2fh\" (UID: \"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e\") " pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:20:31.048014 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:31.047982    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqlb9\""
Sep 05 02:20:31.054205 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:31.054187    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr2fh"
Sep 05 02:20:31.332312 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:31.332235    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jr2fh"]
Sep 05 02:20:31.335853 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:31.335826    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf8a9aa_9c3e_42fb_807e_33aa52e2cd1e.slice/crio-57563fba0f0c840c95a51869a22553508f4e039852f2a601763c0ad167126081 WatchSource:0}: Error finding container 57563fba0f0c840c95a51869a22553508f4e039852f2a601763c0ad167126081: Status 404 returned error can't find the container with id 57563fba0f0c840c95a51869a22553508f4e039852f2a601763c0ad167126081
Sep 05 02:20:31.489185 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:31.489145    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jr2fh" event={"ID":"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e","Type":"ContainerStarted","Data":"57563fba0f0c840c95a51869a22553508f4e039852f2a601763c0ad167126081"}
Sep 05 02:20:31.490603 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:31.490576    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs" event={"ID":"9c8ab5ec-55dd-4ad4-8774-85d5a8893aae","Type":"ContainerStarted","Data":"11968ad9b26c06e66c94e971a2c61e617e9f799514dad36cb61b50026ebd7dd0"}
Sep 05 02:20:31.508636 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:31.508596    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7mqqs" podStartSLOduration=33.827406376 podStartE2EDuration="35.508581864s" totalImagesPullingTime="1.681175488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:31.507518096 +0000 UTC m=+130.934634323" watchObservedRunningTime="2026-09-05 02:20:31.508581864 +0000 UTC m=+130.935698089"
Sep 05 02:20:32.496176 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:32.496084    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jr2fh" event={"ID":"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e","Type":"ContainerStarted","Data":"136a020078d7241ac46db257a07814a813b3c12e35c578300be0efa16bf9009c"}
Sep 05 02:20:32.496176 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:32.496152    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jr2fh" event={"ID":"dbf8a9aa-9c3e-42fb-807e-33aa52e2cd1e","Type":"ContainerStarted","Data":"26c08572a3c185da6f7f13d1309a135302c3f87af628e8be48e74db649065fec"}
Sep 05 02:20:32.512955 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:32.512906    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jr2fh" podStartSLOduration=130.652041536 podStartE2EDuration="2m11.512889744s" totalImagesPullingTime="860.848208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:32.51154408 +0000 UTC m=+131.938660302" watchObservedRunningTime="2026-09-05 02:20:32.512889744 +0000 UTC m=+131.940005969"
Sep 05 02:20:38.457157 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.457121    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mpcbc"]
Sep 05 02:20:38.462707 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.462685    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.464945 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.464897    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:20:38.465242 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.465221    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ld2zj\""
Sep 05 02:20:38.465319 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.465268    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:20:38.466231 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.466212    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:20:38.466314 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.466210    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:20:38.599061 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599035    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e976bb-df8e-4d40-baa7-08bcee923fa7-metrics-client-ca\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599190 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599092    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599242 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-textfile\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599294 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599294 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599288    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-root\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599375 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-tls\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599412 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599381    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-wtmp\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599465 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-sys\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.599465 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.599439    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwsxw\" (UniqueName: \"kubernetes.io/projected/61e976bb-df8e-4d40-baa7-08bcee923fa7-kube-api-access-pwsxw\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.699916 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.699890    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.699916 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.699919    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-root\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.699944    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-tls\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.699970    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-wtmp\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-sys\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700016    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-root\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwsxw\" (UniqueName: \"kubernetes.io/projected/61e976bb-df8e-4d40-baa7-08bcee923fa7-kube-api-access-pwsxw\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700068    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e976bb-df8e-4d40-baa7-08bcee923fa7-metrics-client-ca\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:38.700107    2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:20:38.700123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700503 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-textfile\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700503 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:38.700173    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-tls podName:61e976bb-df8e-4d40-baa7-08bcee923fa7 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:39.2001511 +0000 UTC m=+138.627267318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-tls") pod "node-exporter-mpcbc" (UID: "61e976bb-df8e-4d40-baa7-08bcee923fa7") : secret "node-exporter-tls" not found
Sep 05 02:20:38.700503 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-wtmp\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700503 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700442    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-textfile\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700503 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700486    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61e976bb-df8e-4d40-baa7-08bcee923fa7-sys\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700845 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700827    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e976bb-df8e-4d40-baa7-08bcee923fa7-metrics-client-ca\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.700901 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.700846    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.702920 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.702890    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:38.708838 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:38.708793    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwsxw\" (UniqueName: \"kubernetes.io/projected/61e976bb-df8e-4d40-baa7-08bcee923fa7-kube-api-access-pwsxw\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:39.203666 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:39.203633    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-tls\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:39.206126 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:39.206094    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61e976bb-df8e-4d40-baa7-08bcee923fa7-node-exporter-tls\") pod \"node-exporter-mpcbc\" (UID: \"61e976bb-df8e-4d40-baa7-08bcee923fa7\") " pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:39.372424 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:39.372395    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpcbc"
Sep 05 02:20:39.386448 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:20:39.386422    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e976bb_df8e_4d40_baa7_08bcee923fa7.slice/crio-f2f6b0cf122363e79ba365fb074251694496de8cf7937f63a9dba36b8e96143d WatchSource:0}: Error finding container f2f6b0cf122363e79ba365fb074251694496de8cf7937f63a9dba36b8e96143d: Status 404 returned error can't find the container with id f2f6b0cf122363e79ba365fb074251694496de8cf7937f63a9dba36b8e96143d
Sep 05 02:20:39.521923 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:39.521852    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpcbc" event={"ID":"61e976bb-df8e-4d40-baa7-08bcee923fa7","Type":"ContainerStarted","Data":"f2f6b0cf122363e79ba365fb074251694496de8cf7937f63a9dba36b8e96143d"}
Sep 05 02:20:41.529438 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:41.529402    2568 generic.go:356] "Generic (PLEG): container finished" podID="61e976bb-df8e-4d40-baa7-08bcee923fa7" containerID="2fbb5cf77e21dbf6aeb9c5b4dbe8850acbe2cbb94c61a7c4ac55e656a6a20754" exitCode=0
Sep 05 02:20:41.529820 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:41.529494    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpcbc" event={"ID":"61e976bb-df8e-4d40-baa7-08bcee923fa7","Type":"ContainerDied","Data":"2fbb5cf77e21dbf6aeb9c5b4dbe8850acbe2cbb94c61a7c4ac55e656a6a20754"}
Sep 05 02:20:42.535412 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:42.535384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpcbc" event={"ID":"61e976bb-df8e-4d40-baa7-08bcee923fa7","Type":"ContainerStarted","Data":"10d60847bc52943446854be0cd78bd67975a850539d72c63e28994efb38715ae"}
Sep 05 02:20:42.535765 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:42.535434    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpcbc" event={"ID":"61e976bb-df8e-4d40-baa7-08bcee923fa7","Type":"ContainerStarted","Data":"62746ff350ed68afc8b541cd038a353a7e068688c5ed77977950147cf09248a6"}
Sep 05 02:20:42.555644 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:42.555599    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mpcbc" podStartSLOduration=3.395387549 podStartE2EDuration="4.555584428s" totalImagesPullingTime="1.093646118s" totalInitContainerRuntime="66.550761ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:42.553745642 +0000 UTC m=+141.980861868" watchObservedRunningTime="2026-09-05 02:20:42.555584428 +0000 UTC m=+141.982700652"
Sep 05 02:20:57.912979 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:20:57.912932    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bhwx6" podUID="d7b1d500-6b4b-49e5-8f80-9ffde88c716c"
Sep 05 02:20:58.589928 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:58.589899    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:20:59.842382 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.842350    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9b47fbcd-ljnbh"]
Sep 05 02:20:59.898487 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.898449    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9b47fbcd-ljnbh"]
Sep 05 02:20:59.898647 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.898557    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.901618 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.901593    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:20:59.901853 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.901835    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:20:59.901951 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.901852    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m5q29\""
Sep 05 02:20:59.901951 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.901902    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:20:59.902457 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.902440    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:20:59.902545 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.902471    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:20:59.902696 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.902681    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:20:59.902783 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.902701    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:20:59.907258 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.907238    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 05 02:20:59.945275 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945254    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-service-ca\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.945360 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945298    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-oauth-serving-cert\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.945360 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-config\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.945444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945375    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-serving-cert\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.945444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945397    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-oauth-config\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.945444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8skm\" (UniqueName: \"kubernetes.io/projected/8fd1d85d-9832-410b-932e-c1eb9051e67e-kube-api-access-v8skm\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:20:59.945544 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:20:59.945442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-trusted-ca-bundle\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.046700 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-oauth-serving-cert\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.046791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046706    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-config\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.046791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046728    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-serving-cert\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.046791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046745    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-oauth-config\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.046791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046762    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8skm\" (UniqueName: \"kubernetes.io/projected/8fd1d85d-9832-410b-932e-c1eb9051e67e-kube-api-access-v8skm\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.046791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046777    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-trusted-ca-bundle\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.047028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.046916    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-service-ca\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.047534 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.047508    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-oauth-serving-cert\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.047635 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.047545    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-config\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.047635 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.047596    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-service-ca\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.047999 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.047977    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-trusted-ca-bundle\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.049413 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.049387    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-serving-cert\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.049508 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.049490    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-oauth-config\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.055448 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.055427    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8skm\" (UniqueName: \"kubernetes.io/projected/8fd1d85d-9832-410b-932e-c1eb9051e67e-kube-api-access-v8skm\") pod \"console-5d9b47fbcd-ljnbh\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") " pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.208390 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.208367    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:00.509952 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.509873    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9b47fbcd-ljnbh"]
Sep 05 02:21:00.512984 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:21:00.512961    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd1d85d_9832_410b_932e_c1eb9051e67e.slice/crio-903315bdf40e6d4a65ca77b0675a94aa7591e70c9d7ec80062fcfc457e2c890a WatchSource:0}: Error finding container 903315bdf40e6d4a65ca77b0675a94aa7591e70c9d7ec80062fcfc457e2c890a: Status 404 returned error can't find the container with id 903315bdf40e6d4a65ca77b0675a94aa7591e70c9d7ec80062fcfc457e2c890a
Sep 05 02:21:00.596940 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:00.596912    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b47fbcd-ljnbh" event={"ID":"8fd1d85d-9832-410b-932e-c1eb9051e67e","Type":"ContainerStarted","Data":"903315bdf40e6d4a65ca77b0675a94aa7591e70c9d7ec80062fcfc457e2c890a"}
Sep 05 02:21:02.870117 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:02.870060    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:21:02.873216 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:02.873190    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7b1d500-6b4b-49e5-8f80-9ffde88c716c-metrics-tls\") pod \"dns-default-bhwx6\" (UID: \"d7b1d500-6b4b-49e5-8f80-9ffde88c716c\") " pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:21:03.093188 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:03.093159    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9hcm\""
Sep 05 02:21:03.101437 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:03.101415    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:21:03.609438 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:03.609337    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b47fbcd-ljnbh" event={"ID":"8fd1d85d-9832-410b-932e-c1eb9051e67e","Type":"ContainerStarted","Data":"ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf"}
Sep 05 02:21:03.623147 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:03.623121    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bhwx6"]
Sep 05 02:21:03.626096 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:21:03.626031    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b1d500_6b4b_49e5_8f80_9ffde88c716c.slice/crio-1a53b05411b1a316c9836d28992930d964f93bf9c05f3baac61c896e11d26a23 WatchSource:0}: Error finding container 1a53b05411b1a316c9836d28992930d964f93bf9c05f3baac61c896e11d26a23: Status 404 returned error can't find the container with id 1a53b05411b1a316c9836d28992930d964f93bf9c05f3baac61c896e11d26a23
Sep 05 02:21:03.627191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:03.627153    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9b47fbcd-ljnbh" podStartSLOduration=1.761179561 podStartE2EDuration="4.627139122s" totalImagesPullingTime="2.865959561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:03.62611268 +0000 UTC m=+163.053228905" watchObservedRunningTime="2026-09-05 02:21:03.627139122 +0000 UTC m=+163.054255346"
Sep 05 02:21:04.614400 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:04.614358    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhwx6" event={"ID":"d7b1d500-6b4b-49e5-8f80-9ffde88c716c","Type":"ContainerStarted","Data":"1a53b05411b1a316c9836d28992930d964f93bf9c05f3baac61c896e11d26a23"}
Sep 05 02:21:05.619600 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:05.619563    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhwx6" event={"ID":"d7b1d500-6b4b-49e5-8f80-9ffde88c716c","Type":"ContainerStarted","Data":"ca032dd39897da8b9637fac60a059276dfb8a2412df8bce6c15289709c52a3ab"}
Sep 05 02:21:05.619969 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:05.619622    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhwx6" event={"ID":"d7b1d500-6b4b-49e5-8f80-9ffde88c716c","Type":"ContainerStarted","Data":"05869f8b36e06b6fa20c0529374be8f493ebc89d9798eb01953d95977c5e18ca"}
Sep 05 02:21:05.619969 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:05.619696    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:21:05.639649 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:05.639606    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bhwx6" podStartSLOduration=130.087215349 podStartE2EDuration="2m11.639589411s" totalImagesPullingTime="1.552374062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:05.637965423 +0000 UTC m=+165.065081649" watchObservedRunningTime="2026-09-05 02:21:05.639589411 +0000 UTC m=+165.066705636"
Sep 05 02:21:10.208951 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:10.208911    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:10.209456 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:10.208990    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:10.210407 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:10.210384    2568 patch_prober.go:28] interesting pod/console-5d9b47fbcd-ljnbh container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused" start-of-body=
Sep 05 02:21:10.210473 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:10.210443    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d9b47fbcd-ljnbh" podUID="8fd1d85d-9832-410b-932e-c1eb9051e67e" containerName="console" probeResult="failure" output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused"
Sep 05 02:21:15.625560 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:15.625471    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bhwx6"
Sep 05 02:21:18.098760 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:18.098733    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-7mqqs_9c8ab5ec-55dd-4ad4-8774-85d5a8893aae/cluster-monitoring-operator/0.log"
Sep 05 02:21:19.895215 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:19.895187    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpcbc_61e976bb-df8e-4d40-baa7-08bcee923fa7/init-textfile/0.log"
Sep 05 02:21:20.097214 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:20.097176    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpcbc_61e976bb-df8e-4d40-baa7-08bcee923fa7/node-exporter/0.log"
Sep 05 02:21:20.212894 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:20.212834    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:20.216521 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:20.216502    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:21:20.300817 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:20.300793    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpcbc_61e976bb-df8e-4d40-baa7-08bcee923fa7/kube-rbac-proxy/0.log"
Sep 05 02:21:25.696304 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:25.696266    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d9b47fbcd-ljnbh_8fd1d85d-9832-410b-932e-c1eb9051e67e/console/0.log"
Sep 05 02:21:26.098435 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:26.098353    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhwx6_d7b1d500-6b4b-49e5-8f80-9ffde88c716c/dns/0.log"
Sep 05 02:21:26.295690 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:26.295659    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhwx6_d7b1d500-6b4b-49e5-8f80-9ffde88c716c/kube-rbac-proxy/0.log"
Sep 05 02:21:27.295940 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:27.295915    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z4zs_02284cc8-068b-431e-a769-3ddd6faff502/dns-node-resolver/0.log"
Sep 05 02:21:35.727369 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:35.727342    2568 generic.go:356] "Generic (PLEG): container finished" podID="c6f68797-eb63-4188-a1a6-f27185930675" containerID="58f1113f81703beb5aa161507f38e6ac398a699f6f3aba3051c07178c3e339c9" exitCode=0
Sep 05 02:21:35.727647 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:35.727422    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z" event={"ID":"c6f68797-eb63-4188-a1a6-f27185930675","Type":"ContainerDied","Data":"58f1113f81703beb5aa161507f38e6ac398a699f6f3aba3051c07178c3e339c9"}
Sep 05 02:21:35.727768 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:35.727752    2568 scope.go:117] "RemoveContainer" containerID="58f1113f81703beb5aa161507f38e6ac398a699f6f3aba3051c07178c3e339c9"
Sep 05 02:21:36.732698 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:21:36.732663    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t4t8z" event={"ID":"c6f68797-eb63-4188-a1a6-f27185930675","Type":"ContainerStarted","Data":"8cdf719ece77c6fbecf53274bae947e87e810b5c425c573337393bce16e894c4"}
Sep 05 02:22:07.049254 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.049219    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8654dbf86d-b89v9"]
Sep 05 02:22:07.051180 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.051165    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.062661 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.062640    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8654dbf86d-b89v9"]
Sep 05 02:22:07.109467 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109438    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-oauth-config\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.109467 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109468    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-serving-cert\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.109663 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109491    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-oauth-serving-cert\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.109663 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109577    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vg6\" (UniqueName: \"kubernetes.io/projected/5788579f-769e-4148-a3da-93c04a01910a-kube-api-access-x4vg6\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.109663 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-service-ca\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.109809 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109686    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-console-config\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.109809 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.109704    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-trusted-ca-bundle\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.210805 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.210765    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-console-config\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.210805 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.210802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-trusted-ca-bundle\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211031 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.210828    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-oauth-config\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211031 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.210843    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-serving-cert\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211031 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.210865    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-oauth-serving-cert\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211031 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.211008    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vg6\" (UniqueName: \"kubernetes.io/projected/5788579f-769e-4148-a3da-93c04a01910a-kube-api-access-x4vg6\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211267 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.211057    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-service-ca\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211590 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.211569    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-oauth-serving-cert\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211714 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.211694    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-service-ca\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211857 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.211837    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-console-config\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.211976 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.211957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-trusted-ca-bundle\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.213548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.213518    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-oauth-config\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.213645 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.213629    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-serving-cert\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.219810 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.219789    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vg6\" (UniqueName: \"kubernetes.io/projected/5788579f-769e-4148-a3da-93c04a01910a-kube-api-access-x4vg6\") pod \"console-8654dbf86d-b89v9\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") " pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.360365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.360278    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:07.631160 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.631131    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8654dbf86d-b89v9"]
Sep 05 02:22:07.635101 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:22:07.635053    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5788579f_769e_4148_a3da_93c04a01910a.slice/crio-65b087ad6588a7f0f0f6c6e4d5dad3ead573ec845eb2017c0125512b0d238236 WatchSource:0}: Error finding container 65b087ad6588a7f0f0f6c6e4d5dad3ead573ec845eb2017c0125512b0d238236: Status 404 returned error can't find the container with id 65b087ad6588a7f0f0f6c6e4d5dad3ead573ec845eb2017c0125512b0d238236
Sep 05 02:22:07.842929 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.842889    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8654dbf86d-b89v9" event={"ID":"5788579f-769e-4148-a3da-93c04a01910a","Type":"ContainerStarted","Data":"1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200"}
Sep 05 02:22:07.843127 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.842976    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8654dbf86d-b89v9" event={"ID":"5788579f-769e-4148-a3da-93c04a01910a","Type":"ContainerStarted","Data":"65b087ad6588a7f0f0f6c6e4d5dad3ead573ec845eb2017c0125512b0d238236"}
Sep 05 02:22:07.859320 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:07.859265    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8654dbf86d-b89v9" podStartSLOduration=0.859246116 podStartE2EDuration="859.246116ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:22:07.858875931 +0000 UTC m=+227.285992156" watchObservedRunningTime="2026-09-05 02:22:07.859246116 +0000 UTC m=+227.286362341"
Sep 05 02:22:17.361138 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:17.361100    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:17.361523 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:17.361158    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:17.366122 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:17.366100    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:17.880138 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:17.880110    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:22:17.923681 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:17.923650    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9b47fbcd-ljnbh"]
Sep 05 02:22:42.949149 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:42.949051    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d9b47fbcd-ljnbh" podUID="8fd1d85d-9832-410b-932e-c1eb9051e67e" containerName="console" containerID="cri-o://ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf" gracePeriod=15
Sep 05 02:22:43.185874 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.185854    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:22:43.281835 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281786    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8skm\" (UniqueName: \"kubernetes.io/projected/8fd1d85d-9832-410b-932e-c1eb9051e67e-kube-api-access-v8skm\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.281835 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281830    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-config\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.282028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281857    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-oauth-serving-cert\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.282028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281882    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-oauth-config\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.282028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281925    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-trusted-ca-bundle\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.282028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281964    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-service-ca\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.282028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.281997    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-serving-cert\") pod \"8fd1d85d-9832-410b-932e-c1eb9051e67e\" (UID: \"8fd1d85d-9832-410b-932e-c1eb9051e67e\") "
Sep 05 02:22:43.282303 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.282266    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:43.282303 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.282278    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-config" (OuterVolumeSpecName: "console-config") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:43.282385 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.282359    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-service-ca" (OuterVolumeSpecName: "service-ca") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:43.282419 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.282380    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:43.284061 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.284037    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:43.284178 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.284133    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd1d85d-9832-410b-932e-c1eb9051e67e-kube-api-access-v8skm" (OuterVolumeSpecName: "kube-api-access-v8skm") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "kube-api-access-v8skm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:22:43.284237 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.284219    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8fd1d85d-9832-410b-932e-c1eb9051e67e" (UID: "8fd1d85d-9832-410b-932e-c1eb9051e67e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:43.383341 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383310    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.383341 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383340    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-oauth-serving-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.383341 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383350    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-oauth-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.383497 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383360    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-trusted-ca-bundle\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.383497 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383369    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd1d85d-9832-410b-932e-c1eb9051e67e-service-ca\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.383497 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383377    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd1d85d-9832-410b-932e-c1eb9051e67e-console-serving-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.383497 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.383385    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8skm\" (UniqueName: \"kubernetes.io/projected/8fd1d85d-9832-410b-932e-c1eb9051e67e-kube-api-access-v8skm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:22:43.968679 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.968650    2568 generic.go:356] "Generic (PLEG): container finished" podID="8fd1d85d-9832-410b-932e-c1eb9051e67e" containerID="ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf" exitCode=0
Sep 05 02:22:43.969060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.968694    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b47fbcd-ljnbh"
Sep 05 02:22:43.969060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.968697    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b47fbcd-ljnbh" event={"ID":"8fd1d85d-9832-410b-932e-c1eb9051e67e","Type":"ContainerDied","Data":"ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf"}
Sep 05 02:22:43.969060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.968812    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b47fbcd-ljnbh" event={"ID":"8fd1d85d-9832-410b-932e-c1eb9051e67e","Type":"ContainerDied","Data":"903315bdf40e6d4a65ca77b0675a94aa7591e70c9d7ec80062fcfc457e2c890a"}
Sep 05 02:22:43.969060 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.968832    2568 scope.go:117] "RemoveContainer" containerID="ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf"
Sep 05 02:22:43.978023 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.978005    2568 scope.go:117] "RemoveContainer" containerID="ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf"
Sep 05 02:22:43.978316 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:22:43.978295    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf\": container with ID starting with ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf not found: ID does not exist" containerID="ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf"
Sep 05 02:22:43.978365 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.978327    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf"} err="failed to get container status \"ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf\": rpc error: code = NotFound desc = could not find container \"ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf\": container with ID starting with ad31faf0c138f513eea44320d6c3f41f7a05e501f8dd13aed102b022359521bf not found: ID does not exist"
Sep 05 02:22:43.988831 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.988807    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9b47fbcd-ljnbh"]
Sep 05 02:22:43.992258 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:43.992235    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9b47fbcd-ljnbh"]
Sep 05 02:22:45.033092 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:22:45.033048    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd1d85d-9832-410b-932e-c1eb9051e67e" path="/var/lib/kubelet/pods/8fd1d85d-9832-410b-932e-c1eb9051e67e/volumes"
Sep 05 02:23:20.919311 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:20.918969    2568 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:23:37.745133 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.745101    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c8f57f8d-rm4bm"]
Sep 05 02:23:37.747408 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.745342    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd1d85d-9832-410b-932e-c1eb9051e67e" containerName="console"
Sep 05 02:23:37.747408 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.745353    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd1d85d-9832-410b-932e-c1eb9051e67e" containerName="console"
Sep 05 02:23:37.747408 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.745395    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd1d85d-9832-410b-932e-c1eb9051e67e" containerName="console"
Sep 05 02:23:37.748261 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.748247    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.758876 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.758851    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8f57f8d-rm4bm"]
Sep 05 02:23:37.831821 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.831798    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-config\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.831951 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.831833    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-oauth-config\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.831951 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.831851    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-serving-cert\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.831951 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.831909    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-oauth-serving-cert\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.832139 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.831954    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-trusted-ca-bundle\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.832139 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.831983    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpmv\" (UniqueName: \"kubernetes.io/projected/26df8af6-bbd2-4f25-af0d-0d9203fac45c-kube-api-access-fcpmv\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.832139 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.832035    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-service-ca\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933183 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933157    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-oauth-config\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933307 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-serving-cert\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933307 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933212    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-oauth-serving-cert\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933406 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933339    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-trusted-ca-bundle\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933406 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpmv\" (UniqueName: \"kubernetes.io/projected/26df8af6-bbd2-4f25-af0d-0d9203fac45c-kube-api-access-fcpmv\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933406 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933390    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-service-ca\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933433    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-config\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.933877 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.933854    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-oauth-serving-cert\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.934025 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.934007    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-service-ca\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.934150 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.934130    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-config\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.934386 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.934370    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-trusted-ca-bundle\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.936354 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.936334    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-oauth-config\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.936427 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.936360    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-serving-cert\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:37.943942 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:37.943920    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpmv\" (UniqueName: \"kubernetes.io/projected/26df8af6-bbd2-4f25-af0d-0d9203fac45c-kube-api-access-fcpmv\") pod \"console-5c8f57f8d-rm4bm\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") " pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:38.055932 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:38.055858    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:38.328129 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:38.328044    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8f57f8d-rm4bm"]
Sep 05 02:23:38.330858 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:23:38.330836    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26df8af6_bbd2_4f25_af0d_0d9203fac45c.slice/crio-cb3ceabe26fa8063eba9b4bd37e06676abfbafb64bff5c65dc403ae906854106 WatchSource:0}: Error finding container cb3ceabe26fa8063eba9b4bd37e06676abfbafb64bff5c65dc403ae906854106: Status 404 returned error can't find the container with id cb3ceabe26fa8063eba9b4bd37e06676abfbafb64bff5c65dc403ae906854106
Sep 05 02:23:39.152176 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:39.152140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8f57f8d-rm4bm" event={"ID":"26df8af6-bbd2-4f25-af0d-0d9203fac45c","Type":"ContainerStarted","Data":"54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4"}
Sep 05 02:23:39.152536 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:39.152193    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8f57f8d-rm4bm" event={"ID":"26df8af6-bbd2-4f25-af0d-0d9203fac45c","Type":"ContainerStarted","Data":"cb3ceabe26fa8063eba9b4bd37e06676abfbafb64bff5c65dc403ae906854106"}
Sep 05 02:23:39.181677 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:39.181626    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c8f57f8d-rm4bm" podStartSLOduration=2.181610857 podStartE2EDuration="2.181610857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:23:39.18067231 +0000 UTC m=+318.607788533" watchObservedRunningTime="2026-09-05 02:23:39.181610857 +0000 UTC m=+318.608727086"
Sep 05 02:23:48.056965 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:48.056925    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:48.057349 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:48.056989    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:48.061510 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:48.061490    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:48.189373 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:48.189347    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:23:48.236328 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:23:48.236301    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8654dbf86d-b89v9"]
Sep 05 02:24:13.255432 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.255301    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8654dbf86d-b89v9" podUID="5788579f-769e-4148-a3da-93c04a01910a" containerName="console" containerID="cri-o://1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200" gracePeriod=15
Sep 05 02:24:13.494540 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.494514    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:24:13.591918 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.591836    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-oauth-serving-cert\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.591918 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.591871    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-serving-cert\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.591918 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.591896    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-console-config\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.592220 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.591926    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-trusted-ca-bundle\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.592220 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.591994    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-oauth-config\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.592220 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.592018    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-service-ca\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.592220 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.592058    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4vg6\" (UniqueName: \"kubernetes.io/projected/5788579f-769e-4148-a3da-93c04a01910a-kube-api-access-x4vg6\") pod \"5788579f-769e-4148-a3da-93c04a01910a\" (UID: \"5788579f-769e-4148-a3da-93c04a01910a\") "
Sep 05 02:24:13.592421 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.592402    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:24:13.592474 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.592444    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-console-config" (OuterVolumeSpecName: "console-config") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:24:13.592560 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.592533    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:24:13.592605 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.592537    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-service-ca" (OuterVolumeSpecName: "service-ca") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:24:13.594191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.594162    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:24:13.594291 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.594225    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:24:13.594450 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.594428    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5788579f-769e-4148-a3da-93c04a01910a-kube-api-access-x4vg6" (OuterVolumeSpecName: "kube-api-access-x4vg6") pod "5788579f-769e-4148-a3da-93c04a01910a" (UID: "5788579f-769e-4148-a3da-93c04a01910a"). InnerVolumeSpecName "kube-api-access-x4vg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:24:13.693444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693406    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vg6\" (UniqueName: \"kubernetes.io/projected/5788579f-769e-4148-a3da-93c04a01910a-kube-api-access-x4vg6\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:13.693444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693435    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-oauth-serving-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:13.693444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693445    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-serving-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:13.693679 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693481    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-console-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:13.693679 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693490    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-trusted-ca-bundle\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:13.693679 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693500    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5788579f-769e-4148-a3da-93c04a01910a-console-oauth-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:13.693679 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:13.693508    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5788579f-769e-4148-a3da-93c04a01910a-service-ca\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:24:14.276128 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.276094    2568 generic.go:356] "Generic (PLEG): container finished" podID="5788579f-769e-4148-a3da-93c04a01910a" containerID="1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200" exitCode=0
Sep 05 02:24:14.276516 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.276135    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8654dbf86d-b89v9"
Sep 05 02:24:14.276516 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.276170    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8654dbf86d-b89v9" event={"ID":"5788579f-769e-4148-a3da-93c04a01910a","Type":"ContainerDied","Data":"1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200"}
Sep 05 02:24:14.276516 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.276219    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8654dbf86d-b89v9" event={"ID":"5788579f-769e-4148-a3da-93c04a01910a","Type":"ContainerDied","Data":"65b087ad6588a7f0f0f6c6e4d5dad3ead573ec845eb2017c0125512b0d238236"}
Sep 05 02:24:14.276516 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.276241    2568 scope.go:117] "RemoveContainer" containerID="1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200"
Sep 05 02:24:14.284851 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.284835    2568 scope.go:117] "RemoveContainer" containerID="1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200"
Sep 05 02:24:14.285154 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:24:14.285122    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200\": container with ID starting with 1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200 not found: ID does not exist" containerID="1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200"
Sep 05 02:24:14.285251 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.285166    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200"} err="failed to get container status \"1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200\": rpc error: code = NotFound desc = could not find container \"1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200\": container with ID starting with 1e1df9c2ed29c32912ed91bac71e2a7d6785329ae9a49ddc59246c5efec2f200 not found: ID does not exist"
Sep 05 02:24:14.296568 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.296546    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8654dbf86d-b89v9"]
Sep 05 02:24:14.299786 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:14.299764    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8654dbf86d-b89v9"]
Sep 05 02:24:15.030052 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:15.030016    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5788579f-769e-4148-a3da-93c04a01910a" path="/var/lib/kubelet/pods/5788579f-769e-4148-a3da-93c04a01910a/volumes"
Sep 05 02:24:44.553559 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.553522    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vqsrg"]
Sep 05 02:24:44.554012 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.553790    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5788579f-769e-4148-a3da-93c04a01910a" containerName="console"
Sep 05 02:24:44.554012 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.553802    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5788579f-769e-4148-a3da-93c04a01910a" containerName="console"
Sep 05 02:24:44.554012 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.553869    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5788579f-769e-4148-a3da-93c04a01910a" containerName="console"
Sep 05 02:24:44.556510 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.556494    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:44.566225 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.566201    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 05 02:24:44.566343 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.566290    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7td29\""
Sep 05 02:24:44.567175 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.567158    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 05 02:24:44.567291 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.567278    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 05 02:24:44.567380 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.567366    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 05 02:24:44.588612 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.588581    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vqsrg"]
Sep 05 02:24:44.629122 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.629069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-certificates\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:44.629263 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.629184    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjs6\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-kube-api-access-bjjs6\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:44.729979 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.729941    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-certificates\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:44.730156 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.729996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjs6\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-kube-api-access-bjjs6\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:44.730156 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:24:44.730110    2568 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 05 02:24:44.730156 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:24:44.730134    2568 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-vqsrg: secret "keda-admission-webhooks-certs" not found
Sep 05 02:24:44.730277 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:24:44.730193    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-certificates podName:889c6af1-2347-4a33-8c2e-a63d7f531ebb nodeName:}" failed. No retries permitted until 2026-09-05 02:24:45.230177459 +0000 UTC m=+384.657293667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-certificates") pod "keda-admission-6c4d864b86-vqsrg" (UID: "889c6af1-2347-4a33-8c2e-a63d7f531ebb") : secret "keda-admission-webhooks-certs" not found
Sep 05 02:24:44.765762 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:44.765735    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjs6\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-kube-api-access-bjjs6\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:45.233747 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:45.233708    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-certificates\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:45.236378 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:45.236355    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/889c6af1-2347-4a33-8c2e-a63d7f531ebb-certificates\") pod \"keda-admission-6c4d864b86-vqsrg\" (UID: \"889c6af1-2347-4a33-8c2e-a63d7f531ebb\") " pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:45.465994 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:45.465947    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:45.761361 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:45.761330    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vqsrg"]
Sep 05 02:24:45.764586 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:24:45.764559    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889c6af1_2347_4a33_8c2e_a63d7f531ebb.slice/crio-1928ee7001f7f83b22506cc904687cd1bcd0b6d2231ffa285a82dc3c1c6de43c WatchSource:0}: Error finding container 1928ee7001f7f83b22506cc904687cd1bcd0b6d2231ffa285a82dc3c1c6de43c: Status 404 returned error can't find the container with id 1928ee7001f7f83b22506cc904687cd1bcd0b6d2231ffa285a82dc3c1c6de43c
Sep 05 02:24:45.765786 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:45.765739    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:24:46.386879 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:46.386841    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg" event={"ID":"889c6af1-2347-4a33-8c2e-a63d7f531ebb","Type":"ContainerStarted","Data":"1928ee7001f7f83b22506cc904687cd1bcd0b6d2231ffa285a82dc3c1c6de43c"}
Sep 05 02:24:48.395550 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:48.395513    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg" event={"ID":"889c6af1-2347-4a33-8c2e-a63d7f531ebb","Type":"ContainerStarted","Data":"415c1ede00bc242b33747284214e6eb9c4ceb2fd7b504e0ad020c492023e4cb5"}
Sep 05 02:24:48.396002 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:48.395650    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:24:48.418040 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:24:48.417990    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg" podStartSLOduration=2.168961292 podStartE2EDuration="4.417977454s" totalImagesPullingTime="2.249016162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:48.41712166 +0000 UTC m=+387.844237886" watchObservedRunningTime="2026-09-05 02:24:48.417977454 +0000 UTC m=+387.845093679"
Sep 05 02:25:09.402010 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:09.401976    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-vqsrg"
Sep 05 02:25:51.069571 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.069491    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"]
Sep 05 02:25:51.072598 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.072583    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.076454 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.076436    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 05 02:25:51.076862 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.076844    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:25:51.077381 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.077192    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xwvnl\""
Sep 05 02:25:51.105290 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.105263    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"]
Sep 05 02:25:51.192645 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.192611    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee3d3d0e-154e-43ce-8748-0162ea502dee-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zxz8r\" (UID: \"ee3d3d0e-154e-43ce-8748-0162ea502dee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.192805 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.192653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhq4m\" (UniqueName: \"kubernetes.io/projected/ee3d3d0e-154e-43ce-8748-0162ea502dee-kube-api-access-fhq4m\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zxz8r\" (UID: \"ee3d3d0e-154e-43ce-8748-0162ea502dee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.293174 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.293140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee3d3d0e-154e-43ce-8748-0162ea502dee-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zxz8r\" (UID: \"ee3d3d0e-154e-43ce-8748-0162ea502dee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.293296 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.293198    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhq4m\" (UniqueName: \"kubernetes.io/projected/ee3d3d0e-154e-43ce-8748-0162ea502dee-kube-api-access-fhq4m\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zxz8r\" (UID: \"ee3d3d0e-154e-43ce-8748-0162ea502dee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.293571 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.293550    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee3d3d0e-154e-43ce-8748-0162ea502dee-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zxz8r\" (UID: \"ee3d3d0e-154e-43ce-8748-0162ea502dee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.316820 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.316790    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhq4m\" (UniqueName: \"kubernetes.io/projected/ee3d3d0e-154e-43ce-8748-0162ea502dee-kube-api-access-fhq4m\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zxz8r\" (UID: \"ee3d3d0e-154e-43ce-8748-0162ea502dee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.381842 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.381823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"
Sep 05 02:25:51.699880 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:51.699803    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r"]
Sep 05 02:25:51.703985 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:25:51.703961    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3d3d0e_154e_43ce_8748_0162ea502dee.slice/crio-44c61bde928acf35d83f38c03a40037dd0a40ce4d3e4eec66b8c10f58e1d8a9a WatchSource:0}: Error finding container 44c61bde928acf35d83f38c03a40037dd0a40ce4d3e4eec66b8c10f58e1d8a9a: Status 404 returned error can't find the container with id 44c61bde928acf35d83f38c03a40037dd0a40ce4d3e4eec66b8c10f58e1d8a9a
Sep 05 02:25:52.614827 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:52.614784    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r" event={"ID":"ee3d3d0e-154e-43ce-8748-0162ea502dee","Type":"ContainerStarted","Data":"44c61bde928acf35d83f38c03a40037dd0a40ce4d3e4eec66b8c10f58e1d8a9a"}
Sep 05 02:25:54.624552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:54.624518    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r" event={"ID":"ee3d3d0e-154e-43ce-8748-0162ea502dee","Type":"ContainerStarted","Data":"69531b906b0995d4cc8423c8b68c796e92781818f4fe256d4f9712d2937358b2"}
Sep 05 02:25:54.651613 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:54.651512    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zxz8r" podStartSLOduration=1.244491461 podStartE2EDuration="3.651491156s" totalImagesPullingTime="2.406999695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:54.650826416 +0000 UTC m=+454.077942641" watchObservedRunningTime="2026-09-05 02:25:54.651491156 +0000 UTC m=+454.078607382"
Sep 05 02:25:58.008384 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.008347    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"]
Sep 05 02:25:58.011572 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.011557    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.015399 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.015386    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-p7pdz\""
Sep 05 02:25:58.015548 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.015523    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 05 02:25:58.024647 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.024628    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"]
Sep 05 02:25:58.050063 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.050042    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 05 02:25:58.147049 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.147027    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfc8e5c2-e169-4a26-9351-fc3af4d481f8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zxtbz\" (UID: \"dfc8e5c2-e169-4a26-9351-fc3af4d481f8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.147187 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.147055    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplhq\" (UniqueName: \"kubernetes.io/projected/dfc8e5c2-e169-4a26-9351-fc3af4d481f8-kube-api-access-gplhq\") pod \"cert-manager-cainjector-559cdd8c87-zxtbz\" (UID: \"dfc8e5c2-e169-4a26-9351-fc3af4d481f8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.248187 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.248138    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfc8e5c2-e169-4a26-9351-fc3af4d481f8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zxtbz\" (UID: \"dfc8e5c2-e169-4a26-9351-fc3af4d481f8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.248187 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.248196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplhq\" (UniqueName: \"kubernetes.io/projected/dfc8e5c2-e169-4a26-9351-fc3af4d481f8-kube-api-access-gplhq\") pod \"cert-manager-cainjector-559cdd8c87-zxtbz\" (UID: \"dfc8e5c2-e169-4a26-9351-fc3af4d481f8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.258775 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.258710    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplhq\" (UniqueName: \"kubernetes.io/projected/dfc8e5c2-e169-4a26-9351-fc3af4d481f8-kube-api-access-gplhq\") pod \"cert-manager-cainjector-559cdd8c87-zxtbz\" (UID: \"dfc8e5c2-e169-4a26-9351-fc3af4d481f8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.261266 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.261244    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfc8e5c2-e169-4a26-9351-fc3af4d481f8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zxtbz\" (UID: \"dfc8e5c2-e169-4a26-9351-fc3af4d481f8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.320373 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.320334    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"
Sep 05 02:25:58.594017 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.593782    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz"]
Sep 05 02:25:58.596442 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:25:58.596416    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfc8e5c2_e169_4a26_9351_fc3af4d481f8.slice/crio-aeb4adaf890eb4140ae5afa57ed6df62ee0819d28cd433d0f83f7cc44740c49b WatchSource:0}: Error finding container aeb4adaf890eb4140ae5afa57ed6df62ee0819d28cd433d0f83f7cc44740c49b: Status 404 returned error can't find the container with id aeb4adaf890eb4140ae5afa57ed6df62ee0819d28cd433d0f83f7cc44740c49b
Sep 05 02:25:58.642519 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:25:58.642489    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz" event={"ID":"dfc8e5c2-e169-4a26-9351-fc3af4d481f8","Type":"ContainerStarted","Data":"aeb4adaf890eb4140ae5afa57ed6df62ee0819d28cd433d0f83f7cc44740c49b"}
Sep 05 02:26:01.657191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:01.657149    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz" event={"ID":"dfc8e5c2-e169-4a26-9351-fc3af4d481f8","Type":"ContainerStarted","Data":"5fb6555310655929c2e51d7839c7c6b6de982eb997cf05d9eb624bdd0fb382c6"}
Sep 05 02:26:01.706391 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:01.706338    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zxtbz" podStartSLOduration=2.249096191 podStartE2EDuration="4.706317805s" totalImagesPullingTime="2.457221614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:01.7057287 +0000 UTC m=+461.132844930" watchObservedRunningTime="2026-09-05 02:26:01.706317805 +0000 UTC m=+461.133434030"
Sep 05 02:26:03.972587 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:03.972550    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c8f57f8d-rm4bm"]
Sep 05 02:26:06.534703 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.534670    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhrkt"]
Sep 05 02:26:06.537948 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.537931    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.540527 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.540510    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4k2xb\""
Sep 05 02:26:06.549006 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.548985    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhrkt"]
Sep 05 02:26:06.713938 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.713909    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c42c65-a185-44a8-8acd-f9f21ca18e5d-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhrkt\" (UID: \"d9c42c65-a185-44a8-8acd-f9f21ca18e5d\") " pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.714103 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.713953    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt2v\" (UniqueName: \"kubernetes.io/projected/d9c42c65-a185-44a8-8acd-f9f21ca18e5d-kube-api-access-pxt2v\") pod \"cert-manager-55b4d954d5-zhrkt\" (UID: \"d9c42c65-a185-44a8-8acd-f9f21ca18e5d\") " pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.814326 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.814264    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c42c65-a185-44a8-8acd-f9f21ca18e5d-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhrkt\" (UID: \"d9c42c65-a185-44a8-8acd-f9f21ca18e5d\") " pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.814326 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.814300    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt2v\" (UniqueName: \"kubernetes.io/projected/d9c42c65-a185-44a8-8acd-f9f21ca18e5d-kube-api-access-pxt2v\") pod \"cert-manager-55b4d954d5-zhrkt\" (UID: \"d9c42c65-a185-44a8-8acd-f9f21ca18e5d\") " pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.822773 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.822743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c42c65-a185-44a8-8acd-f9f21ca18e5d-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhrkt\" (UID: \"d9c42c65-a185-44a8-8acd-f9f21ca18e5d\") " pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.822866 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.822783    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt2v\" (UniqueName: \"kubernetes.io/projected/d9c42c65-a185-44a8-8acd-f9f21ca18e5d-kube-api-access-pxt2v\") pod \"cert-manager-55b4d954d5-zhrkt\" (UID: \"d9c42c65-a185-44a8-8acd-f9f21ca18e5d\") " pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:06.846767 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:06.846745    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zhrkt"
Sep 05 02:26:07.117697 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:07.117673    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhrkt"]
Sep 05 02:26:07.120410 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:26:07.120382    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c42c65_a185_44a8_8acd_f9f21ca18e5d.slice/crio-5151a66a55dd118cf325a8cb719a018fd4105c4a92e8ddd657a8b1c8ebc16a8f WatchSource:0}: Error finding container 5151a66a55dd118cf325a8cb719a018fd4105c4a92e8ddd657a8b1c8ebc16a8f: Status 404 returned error can't find the container with id 5151a66a55dd118cf325a8cb719a018fd4105c4a92e8ddd657a8b1c8ebc16a8f
Sep 05 02:26:07.680807 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:07.680775    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zhrkt" event={"ID":"d9c42c65-a185-44a8-8acd-f9f21ca18e5d","Type":"ContainerStarted","Data":"6183667d75815c9a64076d84a6d07d41dc10d9e4812725bda26db1f521ca99eb"}
Sep 05 02:26:07.681199 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:07.680817    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zhrkt" event={"ID":"d9c42c65-a185-44a8-8acd-f9f21ca18e5d","Type":"ContainerStarted","Data":"5151a66a55dd118cf325a8cb719a018fd4105c4a92e8ddd657a8b1c8ebc16a8f"}
Sep 05 02:26:07.698305 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:07.698258    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zhrkt" podStartSLOduration=1.698243076 podStartE2EDuration="1.698243076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:07.697314997 +0000 UTC m=+467.124431222" watchObservedRunningTime="2026-09-05 02:26:07.698243076 +0000 UTC m=+467.125359301"
Sep 05 02:26:28.993194 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:28.993131    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c8f57f8d-rm4bm" podUID="26df8af6-bbd2-4f25-af0d-0d9203fac45c" containerName="console" containerID="cri-o://54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4" gracePeriod=15
Sep 05 02:26:29.228066 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.228044    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:26:29.269048 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.268981    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-service-ca\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.269048 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269032    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-oauth-serving-cert\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.269268 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269065    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-oauth-config\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.269268 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269126    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-trusted-ca-bundle\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.269268 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269181    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-config\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.269268 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269204    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcpmv\" (UniqueName: \"kubernetes.io/projected/26df8af6-bbd2-4f25-af0d-0d9203fac45c-kube-api-access-fcpmv\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.269468 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269348    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-service-ca" (OuterVolumeSpecName: "service-ca") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.269543 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269518    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.269586 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269543    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.269586 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.269554    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-config" (OuterVolumeSpecName: "console-config") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.271332 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.271310    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:26:29.271431 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.271408    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26df8af6-bbd2-4f25-af0d-0d9203fac45c-kube-api-access-fcpmv" (OuterVolumeSpecName: "kube-api-access-fcpmv") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "kube-api-access-fcpmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:29.369842 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369810    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-serving-cert\") pod \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\" (UID: \"26df8af6-bbd2-4f25-af0d-0d9203fac45c\") "
Sep 05 02:26:29.369955 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369943    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-oauth-serving-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.369996 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369957    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-oauth-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.369996 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369966    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-trusted-ca-bundle\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.369996 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369974    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-config\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.369996 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369982    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcpmv\" (UniqueName: \"kubernetes.io/projected/26df8af6-bbd2-4f25-af0d-0d9203fac45c-kube-api-access-fcpmv\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.369996 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.369991    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26df8af6-bbd2-4f25-af0d-0d9203fac45c-service-ca\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.371639 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.371621    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26df8af6-bbd2-4f25-af0d-0d9203fac45c" (UID: "26df8af6-bbd2-4f25-af0d-0d9203fac45c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:26:29.470994 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.470971    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26df8af6-bbd2-4f25-af0d-0d9203fac45c-console-serving-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.768576 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.768542    2568 generic.go:356] "Generic (PLEG): container finished" podID="26df8af6-bbd2-4f25-af0d-0d9203fac45c" containerID="54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4" exitCode=0
Sep 05 02:26:29.768721 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.768589    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8f57f8d-rm4bm"
Sep 05 02:26:29.768721 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.768608    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8f57f8d-rm4bm" event={"ID":"26df8af6-bbd2-4f25-af0d-0d9203fac45c","Type":"ContainerDied","Data":"54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4"}
Sep 05 02:26:29.768721 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.768646    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8f57f8d-rm4bm" event={"ID":"26df8af6-bbd2-4f25-af0d-0d9203fac45c","Type":"ContainerDied","Data":"cb3ceabe26fa8063eba9b4bd37e06676abfbafb64bff5c65dc403ae906854106"}
Sep 05 02:26:29.768721 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.768686    2568 scope.go:117] "RemoveContainer" containerID="54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4"
Sep 05 02:26:29.778238 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.778223    2568 scope.go:117] "RemoveContainer" containerID="54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4"
Sep 05 02:26:29.778507 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:26:29.778489    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4\": container with ID starting with 54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4 not found: ID does not exist" containerID="54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4"
Sep 05 02:26:29.778553 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.778515    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4"} err="failed to get container status \"54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4\": rpc error: code = NotFound desc = could not find container \"54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4\": container with ID starting with 54b458b5a20e4ab375c2ed0150e2e30496698e544f928227c9b11bbbb1a621b4 not found: ID does not exist"
Sep 05 02:26:29.791896 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.791874    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c8f57f8d-rm4bm"]
Sep 05 02:26:29.794593 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:29.794573    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c8f57f8d-rm4bm"]
Sep 05 02:26:31.034050 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.034019    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26df8af6-bbd2-4f25-af0d-0d9203fac45c" path="/var/lib/kubelet/pods/26df8af6-bbd2-4f25-af0d-0d9203fac45c/volumes"
Sep 05 02:26:31.960351 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.960317    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"]
Sep 05 02:26:31.960561 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.960549    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26df8af6-bbd2-4f25-af0d-0d9203fac45c" containerName="console"
Sep 05 02:26:31.960605 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.960563    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="26df8af6-bbd2-4f25-af0d-0d9203fac45c" containerName="console"
Sep 05 02:26:31.960639 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.960614    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="26df8af6-bbd2-4f25-af0d-0d9203fac45c" containerName="console"
Sep 05 02:26:31.963332 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.963317    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:31.968515 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.968490    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:26:31.968628 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.968541    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wwndb\""
Sep 05 02:26:31.968628 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.968490    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 05 02:26:31.983816 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.983786    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/50162386-1069-4362-8003-66dde904dfd8-manager-config\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:31.983917 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.983863    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/50162386-1069-4362-8003-66dde904dfd8-metrics-cert\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:31.983917 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.983895    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50162386-1069-4362-8003-66dde904dfd8-cert\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:31.983987 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.983931    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97zm\" (UniqueName: \"kubernetes.io/projected/50162386-1069-4362-8003-66dde904dfd8-kube-api-access-t97zm\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:31.984639 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.984617    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 05 02:26:31.984753 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.984627    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 05 02:26:31.984753 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.984645    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 05 02:26:31.986758 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:31.986737    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"]
Sep 05 02:26:32.084752 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.084720    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/50162386-1069-4362-8003-66dde904dfd8-manager-config\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.085153 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.084804    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/50162386-1069-4362-8003-66dde904dfd8-metrics-cert\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.085153 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.084833    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50162386-1069-4362-8003-66dde904dfd8-cert\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.085153 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.084873    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97zm\" (UniqueName: \"kubernetes.io/projected/50162386-1069-4362-8003-66dde904dfd8-kube-api-access-t97zm\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.085472 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.085449    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/50162386-1069-4362-8003-66dde904dfd8-manager-config\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.087481 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.087458    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/50162386-1069-4362-8003-66dde904dfd8-metrics-cert\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.087547 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.087514    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50162386-1069-4362-8003-66dde904dfd8-cert\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.093663 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.093631    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97zm\" (UniqueName: \"kubernetes.io/projected/50162386-1069-4362-8003-66dde904dfd8-kube-api-access-t97zm\") pod \"lws-controller-manager-54b595bc86-pq97q\" (UID: \"50162386-1069-4362-8003-66dde904dfd8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.271557 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.271477    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:32.551891 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.551814    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"]
Sep 05 02:26:32.554733 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:26:32.554707    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50162386_1069_4362_8003_66dde904dfd8.slice/crio-78ab2371df008ab29322faab501b90233927991d7d6d2d39eb31402b60266d08 WatchSource:0}: Error finding container 78ab2371df008ab29322faab501b90233927991d7d6d2d39eb31402b60266d08: Status 404 returned error can't find the container with id 78ab2371df008ab29322faab501b90233927991d7d6d2d39eb31402b60266d08
Sep 05 02:26:32.783210 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:32.783181    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q" event={"ID":"50162386-1069-4362-8003-66dde904dfd8","Type":"ContainerStarted","Data":"78ab2371df008ab29322faab501b90233927991d7d6d2d39eb31402b60266d08"}
Sep 05 02:26:35.803569 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:35.803526    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q" event={"ID":"50162386-1069-4362-8003-66dde904dfd8","Type":"ContainerStarted","Data":"a40eebe622b3e5a8e6c5a4e89213ed0885933196c0d1ee64d6d6874a357b95a3"}
Sep 05 02:26:35.803934 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:35.803659    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:26:35.822221 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:35.822178    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q" podStartSLOduration=2.464609236 podStartE2EDuration="4.822165743s" totalImagesPullingTime="2.357556507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:35.821183752 +0000 UTC m=+495.248299980" watchObservedRunningTime="2026-09-05 02:26:35.822165743 +0000 UTC m=+495.249281967"
Sep 05 02:26:46.809588 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:26:46.809559    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-pq97q"
Sep 05 02:27:07.117092 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.117037    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"]
Sep 05 02:27:07.125405 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.125384    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.127882 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.127834    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 05 02:27:07.127882 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.127866    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 05 02:27:07.128917 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.128897    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 05 02:27:07.129041 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.129013    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w4nr9\""
Sep 05 02:27:07.129230 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.128962    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 05 02:27:07.130469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.130446    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"]
Sep 05 02:27:07.236869 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.236830    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wjk\" (UniqueName: \"kubernetes.io/projected/7504a7b0-eea3-4056-973e-eb684b34d21b-kube-api-access-54wjk\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.237111 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.236885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7504a7b0-eea3-4056-973e-eb684b34d21b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.237111 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.237022    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7504a7b0-eea3-4056-973e-eb684b34d21b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.338519 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.338482    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wjk\" (UniqueName: \"kubernetes.io/projected/7504a7b0-eea3-4056-973e-eb684b34d21b-kube-api-access-54wjk\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.338720 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.338527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7504a7b0-eea3-4056-973e-eb684b34d21b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.338720 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.338576    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7504a7b0-eea3-4056-973e-eb684b34d21b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.338720 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:27:07.338683    2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 05 02:27:07.338880 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:27:07.338773    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7504a7b0-eea3-4056-973e-eb684b34d21b-plugin-serving-cert podName:7504a7b0-eea3-4056-973e-eb684b34d21b nodeName:}" failed. No retries permitted until 2026-09-05 02:27:07.838750189 +0000 UTC m=+527.265866396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7504a7b0-eea3-4056-973e-eb684b34d21b-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-wcv4l" (UID: "7504a7b0-eea3-4056-973e-eb684b34d21b") : secret "plugin-serving-cert" not found
Sep 05 02:27:07.339275 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.339258    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7504a7b0-eea3-4056-973e-eb684b34d21b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.347547 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.347527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wjk\" (UniqueName: \"kubernetes.io/projected/7504a7b0-eea3-4056-973e-eb684b34d21b-kube-api-access-54wjk\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.842456 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.842391    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7504a7b0-eea3-4056-973e-eb684b34d21b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:07.844956 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:07.844936    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7504a7b0-eea3-4056-973e-eb684b34d21b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wcv4l\" (UID: \"7504a7b0-eea3-4056-973e-eb684b34d21b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:08.035898 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:08.035863    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"
Sep 05 02:27:08.311768 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:08.311739    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l"]
Sep 05 02:27:08.314615 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:27:08.314584    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7504a7b0_eea3_4056_973e_eb684b34d21b.slice/crio-cea2a3b70ee2039f78ec408b5ef0f7484b6e52cfdc03413f2eb2505e686f7908 WatchSource:0}: Error finding container cea2a3b70ee2039f78ec408b5ef0f7484b6e52cfdc03413f2eb2505e686f7908: Status 404 returned error can't find the container with id cea2a3b70ee2039f78ec408b5ef0f7484b6e52cfdc03413f2eb2505e686f7908
Sep 05 02:27:08.936868 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:08.936833    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l" event={"ID":"7504a7b0-eea3-4056-973e-eb684b34d21b","Type":"ContainerStarted","Data":"cea2a3b70ee2039f78ec408b5ef0f7484b6e52cfdc03413f2eb2505e686f7908"}
Sep 05 02:27:13.962476 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:13.962438    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l" event={"ID":"7504a7b0-eea3-4056-973e-eb684b34d21b","Type":"ContainerStarted","Data":"5fba434da5f19bdbf789a924fdeade4d7653e27f5b9611a8bf2ff9be6b09b532"}
Sep 05 02:27:13.980111 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:13.980051    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wcv4l" podStartSLOduration=2.303220825 podStartE2EDuration="6.980036819s" totalImagesPullingTime="4.676815994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:13.979989866 +0000 UTC m=+533.407106091" watchObservedRunningTime="2026-09-05 02:27:13.980036819 +0000 UTC m=+533.407153043"
Sep 05 02:27:50.574933 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.574895    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:27:50.591775 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.591746    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:27:50.591937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.591841    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.594397 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.594370    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 05 02:27:50.675392 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.675357    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:27:50.685191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.685149    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b59fae8-3273-458c-8b82-427b6c8781fe-config-file\") pod \"limitador-limitador-859c4d96d6-bffct\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.685191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.685201    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdzm\" (UniqueName: \"kubernetes.io/projected/0b59fae8-3273-458c-8b82-427b6c8781fe-kube-api-access-dbdzm\") pod \"limitador-limitador-859c4d96d6-bffct\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.785843 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.785800    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b59fae8-3273-458c-8b82-427b6c8781fe-config-file\") pod \"limitador-limitador-859c4d96d6-bffct\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.785843 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.785848    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdzm\" (UniqueName: \"kubernetes.io/projected/0b59fae8-3273-458c-8b82-427b6c8781fe-kube-api-access-dbdzm\") pod \"limitador-limitador-859c4d96d6-bffct\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.786505 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.786481    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b59fae8-3273-458c-8b82-427b6c8781fe-config-file\") pod \"limitador-limitador-859c4d96d6-bffct\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.795618 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.795596    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdzm\" (UniqueName: \"kubernetes.io/projected/0b59fae8-3273-458c-8b82-427b6c8781fe-kube-api-access-dbdzm\") pod \"limitador-limitador-859c4d96d6-bffct\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:50.903925 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:50.903886    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:51.179596 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:51.179509    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:27:51.182565 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:27:51.182534    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b59fae8_3273_458c_8b82_427b6c8781fe.slice/crio-6d8ca4e80ca39c0a07b9c68998244764ae69338d41d49741bab11454d98bac00 WatchSource:0}: Error finding container 6d8ca4e80ca39c0a07b9c68998244764ae69338d41d49741bab11454d98bac00: Status 404 returned error can't find the container with id 6d8ca4e80ca39c0a07b9c68998244764ae69338d41d49741bab11454d98bac00
Sep 05 02:27:52.108773 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:52.108735    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct" event={"ID":"0b59fae8-3273-458c-8b82-427b6c8781fe","Type":"ContainerStarted","Data":"6d8ca4e80ca39c0a07b9c68998244764ae69338d41d49741bab11454d98bac00"}
Sep 05 02:27:54.120893 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:54.120858    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct" event={"ID":"0b59fae8-3273-458c-8b82-427b6c8781fe","Type":"ContainerStarted","Data":"a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73"}
Sep 05 02:27:54.121288 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:54.120976    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:27:54.140263 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:27:54.140210    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct" podStartSLOduration=1.45814512 podStartE2EDuration="4.140192966s" totalImagesPullingTime="2.682047846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:54.139211079 +0000 UTC m=+573.566327306" watchObservedRunningTime="2026-09-05 02:27:54.140192966 +0000 UTC m=+573.567309191"
Sep 05 02:28:05.126299 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:05.126268    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:28:05.534610 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:05.534512    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:28:05.534791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:05.534746    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct" podUID="0b59fae8-3273-458c-8b82-427b6c8781fe" containerName="limitador" containerID="cri-o://a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73" gracePeriod=30
Sep 05 02:28:06.074866 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.074842    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:28:06.086045 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.086020    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbdzm\" (UniqueName: \"kubernetes.io/projected/0b59fae8-3273-458c-8b82-427b6c8781fe-kube-api-access-dbdzm\") pod \"0b59fae8-3273-458c-8b82-427b6c8781fe\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") "
Sep 05 02:28:06.086217 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.086053    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b59fae8-3273-458c-8b82-427b6c8781fe-config-file\") pod \"0b59fae8-3273-458c-8b82-427b6c8781fe\" (UID: \"0b59fae8-3273-458c-8b82-427b6c8781fe\") "
Sep 05 02:28:06.086422 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.086399    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b59fae8-3273-458c-8b82-427b6c8781fe-config-file" (OuterVolumeSpecName: "config-file") pod "0b59fae8-3273-458c-8b82-427b6c8781fe" (UID: "0b59fae8-3273-458c-8b82-427b6c8781fe"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:28:06.088408 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.088386    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b59fae8-3273-458c-8b82-427b6c8781fe-kube-api-access-dbdzm" (OuterVolumeSpecName: "kube-api-access-dbdzm") pod "0b59fae8-3273-458c-8b82-427b6c8781fe" (UID: "0b59fae8-3273-458c-8b82-427b6c8781fe"). InnerVolumeSpecName "kube-api-access-dbdzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:28:06.169339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.169307    2568 generic.go:356] "Generic (PLEG): container finished" podID="0b59fae8-3273-458c-8b82-427b6c8781fe" containerID="a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73" exitCode=0
Sep 05 02:28:06.169809 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.169355    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct"
Sep 05 02:28:06.169809 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.169360    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct" event={"ID":"0b59fae8-3273-458c-8b82-427b6c8781fe","Type":"ContainerDied","Data":"a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73"}
Sep 05 02:28:06.169809 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.169483    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bffct" event={"ID":"0b59fae8-3273-458c-8b82-427b6c8781fe","Type":"ContainerDied","Data":"6d8ca4e80ca39c0a07b9c68998244764ae69338d41d49741bab11454d98bac00"}
Sep 05 02:28:06.169809 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.169516    2568 scope.go:117] "RemoveContainer" containerID="a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73"
Sep 05 02:28:06.178249 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.178232    2568 scope.go:117] "RemoveContainer" containerID="a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73"
Sep 05 02:28:06.178515 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:28:06.178498    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73\": container with ID starting with a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73 not found: ID does not exist" containerID="a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73"
Sep 05 02:28:06.178558 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.178524    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73"} err="failed to get container status \"a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73\": rpc error: code = NotFound desc = could not find container \"a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73\": container with ID starting with a057173d5c5cefe054c2f4a5515c177cfd319af323cac4607ac73822b790ea73 not found: ID does not exist"
Sep 05 02:28:06.187355 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.187333    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbdzm\" (UniqueName: \"kubernetes.io/projected/0b59fae8-3273-458c-8b82-427b6c8781fe-kube-api-access-dbdzm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:28:06.187451 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.187357    2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b59fae8-3273-458c-8b82-427b6c8781fe-config-file\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:28:06.197384 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.197358    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:28:06.201564 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:06.201544    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bffct"]
Sep 05 02:28:07.030089 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:07.030032    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b59fae8-3273-458c-8b82-427b6c8781fe" path="/var/lib/kubelet/pods/0b59fae8-3273-458c-8b82-427b6c8781fe/volumes"
Sep 05 02:28:34.646268 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.646230    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"]
Sep 05 02:28:34.647483 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.647458    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b59fae8-3273-458c-8b82-427b6c8781fe" containerName="limitador"
Sep 05 02:28:34.647630 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.647619    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b59fae8-3273-458c-8b82-427b6c8781fe" containerName="limitador"
Sep 05 02:28:34.647878 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.647866    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b59fae8-3273-458c-8b82-427b6c8781fe" containerName="limitador"
Sep 05 02:28:34.656142 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.656106    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.658849 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.658826    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 05 02:28:34.658982 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.658826    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 05 02:28:34.659795 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.659777    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j9gq7\""
Sep 05 02:28:34.660422 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.660392    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 05 02:28:34.660422 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.660518    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 05 02:28:34.660422 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.660559    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 02:28:34.661656 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.661635    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"]
Sep 05 02:28:34.664269 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.664252    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 02:28:34.710767 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.710735    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91744050-b528-43d9-8bdb-8a2e67993ccb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.711011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.710790    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.711011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.710851    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.711011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.710875    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdr2\" (UniqueName: \"kubernetes.io/projected/91744050-b528-43d9-8bdb-8a2e67993ccb-kube-api-access-mrdr2\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.711011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.710905    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.711011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.710984    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.711349 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.711021    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.811810 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811769    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.811810 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811809    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdr2\" (UniqueName: \"kubernetes.io/projected/91744050-b528-43d9-8bdb-8a2e67993ccb-kube-api-access-mrdr2\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.812055 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811833    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.812055 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811866    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.812055 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811887    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.812055 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811914    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91744050-b528-43d9-8bdb-8a2e67993ccb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.812055 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.811940    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.812646 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.812617    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.814410 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.814378    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91744050-b528-43d9-8bdb-8a2e67993ccb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.814566 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.814547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.814632 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.814605    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.814685 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.814653    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.826464 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.826439    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdr2\" (UniqueName: \"kubernetes.io/projected/91744050-b528-43d9-8bdb-8a2e67993ccb-kube-api-access-mrdr2\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.828933 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.828910    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91744050-b528-43d9-8bdb-8a2e67993ccb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tc4x8\" (UID: \"91744050-b528-43d9-8bdb-8a2e67993ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:34.966694 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:34.966597    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:35.246708 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:35.246494    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"]
Sep 05 02:28:35.249413 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:28:35.249385    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91744050_b528_43d9_8bdb_8a2e67993ccb.slice/crio-bfa10d58fd15d57ad4743d5b2e80ba830a4c1d956a3f21f4f8c6aa98f8a6c00a WatchSource:0}: Error finding container bfa10d58fd15d57ad4743d5b2e80ba830a4c1d956a3f21f4f8c6aa98f8a6c00a: Status 404 returned error can't find the container with id bfa10d58fd15d57ad4743d5b2e80ba830a4c1d956a3f21f4f8c6aa98f8a6c00a
Sep 05 02:28:35.289530 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:35.289493    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8" event={"ID":"91744050-b528-43d9-8bdb-8a2e67993ccb","Type":"ContainerStarted","Data":"bfa10d58fd15d57ad4743d5b2e80ba830a4c1d956a3f21f4f8c6aa98f8a6c00a"}
Sep 05 02:28:37.608028 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:37.607989    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 05 02:28:37.608283 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:37.608060    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 05 02:28:38.305597 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:38.305565    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8" event={"ID":"91744050-b528-43d9-8bdb-8a2e67993ccb","Type":"ContainerStarted","Data":"d9e4e1c726f6b41e1876d9583ceab4eca1a0775efb5286e260aed9c855e59f93"}
Sep 05 02:28:38.307836 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:38.305773    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:28:38.307836 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:38.307480    2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-tc4x8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 02:28:38.307836 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:38.307533    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8" podUID="91744050-b528-43d9-8bdb-8a2e67993ccb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:28:38.328386 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:38.328336    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8" podStartSLOduration=1.97182398 podStartE2EDuration="4.328321647s" totalImagesPullingTime="2.356497667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:38.326481819 +0000 UTC m=+617.753598056" watchObservedRunningTime="2026-09-05 02:28:38.328321647 +0000 UTC m=+617.755437872"
Sep 05 02:28:39.310508 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:28:39.310477    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tc4x8"
Sep 05 02:35:03.453599 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.453562    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 05 02:35:03.456753 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.456724    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.459178 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.459154    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 05 02:35:03.459605 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.459573    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:35:03.460701 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.460676    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 05 02:35:03.460701 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.460692    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-r2g49\""
Sep 05 02:35:03.460864 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.460692    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 05 02:35:03.460864 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.460721    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gcxjp\""
Sep 05 02:35:03.460864 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.460693    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 05 02:35:03.469029 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.469008    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 05 02:35:03.528649 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.528788 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528661    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.528788 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528723    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.528788 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528752    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.528965 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.528965 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528875    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/182050e0-ff1c-4765-936d-4f6293218060-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.528965 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528914    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182050e0-ff1c-4765-936d-4f6293218060-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.529094 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.528991    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltv96\" (UniqueName: \"kubernetes.io/projected/182050e0-ff1c-4765-936d-4f6293218060-kube-api-access-ltv96\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.629874 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.629847    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltv96\" (UniqueName: \"kubernetes.io/projected/182050e0-ff1c-4765-936d-4f6293218060-kube-api-access-ltv96\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.629998 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.629877    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.629998 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.629896    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.629998 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.629950    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.629998 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.629995    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630211 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630020    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630211 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630042    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/182050e0-ff1c-4765-936d-4f6293218060-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630211 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630194    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182050e0-ff1c-4765-936d-4f6293218060-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630314 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630268    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630358 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630342    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630572 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630549    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630572 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630566    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/182050e0-ff1c-4765-936d-4f6293218060-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.630689 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.630565    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.632420 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.632401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.632651 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.632635    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182050e0-ff1c-4765-936d-4f6293218060-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.638173 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.638152    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltv96\" (UniqueName: \"kubernetes.io/projected/182050e0-ff1c-4765-936d-4f6293218060-kube-api-access-ltv96\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:03.766506 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:03.766437    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:35:04.039306 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:04.039229    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 05 02:35:04.042091 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:35:04.042046    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182050e0_ff1c_4765_936d_4f6293218060.slice/crio-088f71d963dc766742ab37dd65dccc519fcd5da63d68c117ed63ed46fbee26d9 WatchSource:0}: Error finding container 088f71d963dc766742ab37dd65dccc519fcd5da63d68c117ed63ed46fbee26d9: Status 404 returned error can't find the container with id 088f71d963dc766742ab37dd65dccc519fcd5da63d68c117ed63ed46fbee26d9
Sep 05 02:35:04.044129 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:04.044110    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:35:04.847699 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:04.847665    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"182050e0-ff1c-4765-936d-4f6293218060","Type":"ContainerStarted","Data":"088f71d963dc766742ab37dd65dccc519fcd5da63d68c117ed63ed46fbee26d9"}
Sep 05 02:35:07.863351 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:07.863314    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"182050e0-ff1c-4765-936d-4f6293218060","Type":"ContainerStarted","Data":"845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66"}
Sep 05 02:35:09.876117 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:09.876050    2568 generic.go:356] "Generic (PLEG): container finished" podID="182050e0-ff1c-4765-936d-4f6293218060" containerID="845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66" exitCode=0
Sep 05 02:35:09.876489 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:09.876136    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"182050e0-ff1c-4765-936d-4f6293218060","Type":"ContainerDied","Data":"845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66"}
Sep 05 02:35:39.031121 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:39.031059    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"182050e0-ff1c-4765-936d-4f6293218060","Type":"ContainerStarted","Data":"6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786"}
Sep 05 02:35:39.056272 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:35:39.056220    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.220158986 podStartE2EDuration="36.056205309s" totalImagesPullingTime="31.767924645s" totalInitContainerRuntime="2.068121678s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:39.05366503 +0000 UTC m=+1038.480781268" watchObservedRunningTime="2026-09-05 02:35:39.056205309 +0000 UTC m=+1038.483321533"
Sep 05 02:37:33.897752 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:33.897716    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 05 02:37:48.920754 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:48.920643    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="182050e0-ff1c-4765-936d-4f6293218060" containerName="main" containerID="cri-o://6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786" gracePeriod=45
Sep 05 02:37:49.568661 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.568633    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:37:49.577830 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.577808    2568 generic.go:356] "Generic (PLEG): container finished" podID="182050e0-ff1c-4765-936d-4f6293218060" containerID="6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786" exitCode=0
Sep 05 02:37:49.577944 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.577853    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 05 02:37:49.577944 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.577883    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"182050e0-ff1c-4765-936d-4f6293218060","Type":"ContainerDied","Data":"6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786"}
Sep 05 02:37:49.577944 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.577938    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"182050e0-ff1c-4765-936d-4f6293218060","Type":"ContainerDied","Data":"088f71d963dc766742ab37dd65dccc519fcd5da63d68c117ed63ed46fbee26d9"}
Sep 05 02:37:49.578058 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.577971    2568 scope.go:117] "RemoveContainer" containerID="6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786"
Sep 05 02:37:49.588437 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.588421    2568 scope.go:117] "RemoveContainer" containerID="845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66"
Sep 05 02:37:49.595903 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.595888    2568 scope.go:117] "RemoveContainer" containerID="6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786"
Sep 05 02:37:49.596172 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:37:49.596156    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786\": container with ID starting with 6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786 not found: ID does not exist" containerID="6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786"
Sep 05 02:37:49.596221 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.596182    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786"} err="failed to get container status \"6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786\": rpc error: code = NotFound desc = could not find container \"6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786\": container with ID starting with 6e141631df5e839cd0728a3e5bc9100612ecc4b3d2bd1defc5d34b6af2c71786 not found: ID does not exist"
Sep 05 02:37:49.596221 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.596200    2568 scope.go:117] "RemoveContainer" containerID="845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66"
Sep 05 02:37:49.596431 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:37:49.596413    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66\": container with ID starting with 845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66 not found: ID does not exist" containerID="845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66"
Sep 05 02:37:49.596488 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.596441    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66"} err="failed to get container status \"845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66\": rpc error: code = NotFound desc = could not find container \"845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66\": container with ID starting with 845ddc3055ee8ff257c3a13b1bf46d2f4eb991550ad41f1e2119de98f8da2c66 not found: ID does not exist"
Sep 05 02:37:49.681271 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681217    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-dshm\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681271 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681257    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182050e0-ff1c-4765-936d-4f6293218060-tls-certs\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681392 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681279    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-home\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681392 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681301    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-tmp-dir\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681392 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681345    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/182050e0-ff1c-4765-936d-4f6293218060-cabundle-cert\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681392 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681388    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-kserve-provision-location\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681428    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-model-cache\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681457    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltv96\" (UniqueName: \"kubernetes.io/projected/182050e0-ff1c-4765-936d-4f6293218060-kube-api-access-ltv96\") pod \"182050e0-ff1c-4765-936d-4f6293218060\" (UID: \"182050e0-ff1c-4765-936d-4f6293218060\") "
Sep 05 02:37:49.681913 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681883    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182050e0-ff1c-4765-936d-4f6293218060-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:37:49.682041 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.681932    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-home" (OuterVolumeSpecName: "home") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:49.682184 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.682155    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-model-cache" (OuterVolumeSpecName: "model-cache") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:49.683517 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.683493    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-dshm" (OuterVolumeSpecName: "dshm") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:49.684042 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.684022    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182050e0-ff1c-4765-936d-4f6293218060-kube-api-access-ltv96" (OuterVolumeSpecName: "kube-api-access-ltv96") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "kube-api-access-ltv96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:37:49.684202 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.684050    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182050e0-ff1c-4765-936d-4f6293218060-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:37:49.693820 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.693794    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:49.707763 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.707738    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "182050e0-ff1c-4765-936d-4f6293218060" (UID: "182050e0-ff1c-4765-936d-4f6293218060"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:49.782688 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782664    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782688 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782686    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782696    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltv96\" (UniqueName: \"kubernetes.io/projected/182050e0-ff1c-4765-936d-4f6293218060-kube-api-access-ltv96\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782704    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782712    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182050e0-ff1c-4765-936d-4f6293218060-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782721    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782728    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182050e0-ff1c-4765-936d-4f6293218060-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.782806 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.782735    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/182050e0-ff1c-4765-936d-4f6293218060-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:37:49.900314 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.900281    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 05 02:37:49.904604 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:49.904580    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 05 02:37:51.030943 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:37:51.030909    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182050e0-ff1c-4765-936d-4f6293218060" path="/var/lib/kubelet/pods/182050e0-ff1c-4765-936d-4f6293218060/volumes"
Sep 05 02:43:15.247869 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.247826    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"]
Sep 05 02:43:15.248469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.248271    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182050e0-ff1c-4765-936d-4f6293218060" containerName="storage-initializer"
Sep 05 02:43:15.248469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.248291    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="182050e0-ff1c-4765-936d-4f6293218060" containerName="storage-initializer"
Sep 05 02:43:15.248469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.248309    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182050e0-ff1c-4765-936d-4f6293218060" containerName="main"
Sep 05 02:43:15.248469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.248318    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="182050e0-ff1c-4765-936d-4f6293218060" containerName="main"
Sep 05 02:43:15.248469 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.248399    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="182050e0-ff1c-4765-936d-4f6293218060" containerName="main"
Sep 05 02:43:15.251306 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.251286    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.254693 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.254673    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-99qr6\""
Sep 05 02:43:15.255416 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.255399    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-xm9jv\""
Sep 05 02:43:15.255517 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.255469    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 05 02:43:15.255768 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.255751    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 05 02:43:15.255850 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.255839    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 05 02:43:15.271531 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.271511    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"]
Sep 05 02:43:15.343231 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.343189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdhs\" (UniqueName: \"kubernetes.io/projected/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-kube-api-access-dzdhs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.343403 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.343271    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.443902 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.443869    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.444058 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.443918    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdhs\" (UniqueName: \"kubernetes.io/projected/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-kube-api-access-dzdhs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.446552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.446527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.452580 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.452559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdhs\" (UniqueName: \"kubernetes.io/projected/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-kube-api-access-dzdhs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.562246 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.562173    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:15.844021 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.843935    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"]
Sep 05 02:43:15.847311 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:43:15.847281    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c3b6ac_03b5_45b3_8fbf_b28c6f578c97.slice/crio-1d73627e045d34694568491b7e825a63d02e24f0a8d2910261d0a52338c6c5c8 WatchSource:0}: Error finding container 1d73627e045d34694568491b7e825a63d02e24f0a8d2910261d0a52338c6c5c8: Status 404 returned error can't find the container with id 1d73627e045d34694568491b7e825a63d02e24f0a8d2910261d0a52338c6c5c8
Sep 05 02:43:15.849842 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.849827    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:43:15.905112 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:15.905060    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" event={"ID":"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97","Type":"ContainerStarted","Data":"1d73627e045d34694568491b7e825a63d02e24f0a8d2910261d0a52338c6c5c8"}
Sep 05 02:43:18.920725 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:18.920686    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" event={"ID":"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97","Type":"ContainerStarted","Data":"1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968"}
Sep 05 02:43:18.921144 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:18.920929    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:43:18.940571 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:18.940527    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podStartSLOduration=1.503410975 podStartE2EDuration="3.940513788s" totalImagesPullingTime="2.437102813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:18.939147865 +0000 UTC m=+1498.366264093" watchObservedRunningTime="2026-09-05 02:43:18.940513788 +0000 UTC m=+1498.367630013"
Sep 05 02:43:49.772011 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:49.771922    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"]
Sep 05 02:43:49.773642 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:49.773607    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:43:59.774037 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:43:59.774000    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:44:04.791787 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:04.791743    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main" containerID="cri-o://1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968" gracePeriod=45
Sep 05 02:44:09.773779 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:09.773743    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:44:19.774039 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:19.773995    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:44:29.774115 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:29.774051    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:44:35.039127 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.039101    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:44:35.146518 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.146488    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzdhs\" (UniqueName: \"kubernetes.io/projected/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-kube-api-access-dzdhs\") pod \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") "
Sep 05 02:44:35.146773 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.146538    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-tls-certs\") pod \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\" (UID: \"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97\") "
Sep 05 02:44:35.148785 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.148764    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" (UID: "d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:44:35.148918 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.148897    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-kube-api-access-dzdhs" (OuterVolumeSpecName: "kube-api-access-dzdhs") pod "d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" (UID: "d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97"). InnerVolumeSpecName "kube-api-access-dzdhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:44:35.222325 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.222300    2568 generic.go:356] "Generic (PLEG): container finished" podID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerID="1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968" exitCode=0
Sep 05 02:44:35.222444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.222340    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" event={"ID":"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97","Type":"ContainerDied","Data":"1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968"}
Sep 05 02:44:35.222444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.222345    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"
Sep 05 02:44:35.222444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.222379    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk" event={"ID":"d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97","Type":"ContainerDied","Data":"1d73627e045d34694568491b7e825a63d02e24f0a8d2910261d0a52338c6c5c8"}
Sep 05 02:44:35.222444 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.222399    2568 scope.go:117] "RemoveContainer" containerID="1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968"
Sep 05 02:44:35.231390 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.231373    2568 scope.go:117] "RemoveContainer" containerID="1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968"
Sep 05 02:44:35.231686 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:44:35.231667    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968\": container with ID starting with 1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968 not found: ID does not exist" containerID="1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968"
Sep 05 02:44:35.231733 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.231695    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968"} err="failed to get container status \"1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968\": rpc error: code = NotFound desc = could not find container \"1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968\": container with ID starting with 1eb71dcaadb22b14213013e9c56c7aaac992774e8229728480d2fe7261556968 not found: ID does not exist"
Sep 05 02:44:35.243278 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.243248    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"]
Sep 05 02:44:35.246773 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.246747    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56fsxk"]
Sep 05 02:44:35.247709 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.247691    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdhs\" (UniqueName: \"kubernetes.io/projected/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-kube-api-access-dzdhs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:44:35.247755 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:35.247713    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:44:37.030018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:37.029988    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" path="/var/lib/kubelet/pods/d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97/volumes"
Sep 05 02:44:48.547900 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.547855    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"]
Sep 05 02:44:48.548602 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.548540    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main"
Sep 05 02:44:48.548602 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.548568    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main"
Sep 05 02:44:48.548807 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.548740    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c3b6ac-03b5-45b3-8fbf-b28c6f578c97" containerName="main"
Sep 05 02:44:48.553926 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.553905    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.556499 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.556478    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-9hmh8\""
Sep 05 02:44:48.556600 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.556489    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 05 02:44:48.557443 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.557413    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 05 02:44:48.557622 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.557604    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-xgznt\""
Sep 05 02:44:48.557688 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.557674    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 05 02:44:48.563260 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.563242    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"]
Sep 05 02:44:48.744500 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.744472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.744638 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.744510    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87vw\" (UniqueName: \"kubernetes.io/projected/b7a753ea-9efe-4133-931a-e8edfd638d95-kube-api-access-t87vw\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.744638 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.744549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a753ea-9efe-4133-931a-e8edfd638d95-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.744638 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.744616    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.845431 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.845342    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87vw\" (UniqueName: \"kubernetes.io/projected/b7a753ea-9efe-4133-931a-e8edfd638d95-kube-api-access-t87vw\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.845431 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.845393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a753ea-9efe-4133-931a-e8edfd638d95-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.845431 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.845436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.845644 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.845462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.845774 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.845758    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.845826 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.845806    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.847939 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.847920    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a753ea-9efe-4133-931a-e8edfd638d95-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.853778 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.853760    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87vw\" (UniqueName: \"kubernetes.io/projected/b7a753ea-9efe-4133-931a-e8edfd638d95-kube-api-access-t87vw\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:48.864917 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:48.864898    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:49.136841 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:49.136821    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"]
Sep 05 02:44:49.138953 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:44:49.138922    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a753ea_9efe_4133_931a_e8edfd638d95.slice/crio-430c81a5704ee406b294637e7ffbed538ad435426f692c48995272d077aa50f4 WatchSource:0}: Error finding container 430c81a5704ee406b294637e7ffbed538ad435426f692c48995272d077aa50f4: Status 404 returned error can't find the container with id 430c81a5704ee406b294637e7ffbed538ad435426f692c48995272d077aa50f4
Sep 05 02:44:49.282294 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:49.282256    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" event={"ID":"b7a753ea-9efe-4133-931a-e8edfd638d95","Type":"ContainerStarted","Data":"d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea"}
Sep 05 02:44:49.282450 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:49.282306    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" event={"ID":"b7a753ea-9efe-4133-931a-e8edfd638d95","Type":"ContainerStarted","Data":"430c81a5704ee406b294637e7ffbed538ad435426f692c48995272d077aa50f4"}
Sep 05 02:44:49.282450 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:49.282356    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:44:49.298811 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:44:49.298760    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" podStartSLOduration=1.298742828 podStartE2EDuration="1.298742828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:44:49.298523706 +0000 UTC m=+1588.725639932" watchObservedRunningTime="2026-09-05 02:44:49.298742828 +0000 UTC m=+1588.725859054"
Sep 05 02:45:20.287366 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:20.287335    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:45:21.528759 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:21.528728    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"]
Sep 05 02:45:30.287599 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:30.287557    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:36.548305 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:36.548242    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main" containerID="cri-o://d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea" gracePeriod=45
Sep 05 02:45:40.288265 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:40.288219    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:50.288286 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:50.288242    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:50.288678 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:45:50.288344    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:46:00.288471 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:00.288431    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:46:06.795396 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.795367    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:46:06.928550 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.928518    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t87vw\" (UniqueName: \"kubernetes.io/projected/b7a753ea-9efe-4133-931a-e8edfd638d95-kube-api-access-t87vw\") pod \"b7a753ea-9efe-4133-931a-e8edfd638d95\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") "
Sep 05 02:46:06.928732 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.928555    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a753ea-9efe-4133-931a-e8edfd638d95-tls-certs\") pod \"b7a753ea-9efe-4133-931a-e8edfd638d95\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") "
Sep 05 02:46:06.928732 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.928589    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-tokenizer-cache\") pod \"b7a753ea-9efe-4133-931a-e8edfd638d95\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") "
Sep 05 02:46:06.928732 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.928625    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-cachi2-cache\") pod \"b7a753ea-9efe-4133-931a-e8edfd638d95\" (UID: \"b7a753ea-9efe-4133-931a-e8edfd638d95\") "
Sep 05 02:46:06.928917 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.928838    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b7a753ea-9efe-4133-931a-e8edfd638d95" (UID: "b7a753ea-9efe-4133-931a-e8edfd638d95"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:46:06.928967 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.928928    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "b7a753ea-9efe-4133-931a-e8edfd638d95" (UID: "b7a753ea-9efe-4133-931a-e8edfd638d95"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:46:06.930962 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.930934    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a753ea-9efe-4133-931a-e8edfd638d95-kube-api-access-t87vw" (OuterVolumeSpecName: "kube-api-access-t87vw") pod "b7a753ea-9efe-4133-931a-e8edfd638d95" (UID: "b7a753ea-9efe-4133-931a-e8edfd638d95"). InnerVolumeSpecName "kube-api-access-t87vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:46:06.930962 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:06.930944    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a753ea-9efe-4133-931a-e8edfd638d95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7a753ea-9efe-4133-931a-e8edfd638d95" (UID: "b7a753ea-9efe-4133-931a-e8edfd638d95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:46:07.029085 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.029040    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87vw\" (UniqueName: \"kubernetes.io/projected/b7a753ea-9efe-4133-931a-e8edfd638d95-kube-api-access-t87vw\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:46:07.029085 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.029064    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a753ea-9efe-4133-931a-e8edfd638d95-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:46:07.029085 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.029091    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-tokenizer-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:46:07.029298 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.029099    2568 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a753ea-9efe-4133-931a-e8edfd638d95-cachi2-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:46:07.600131 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.600061    2568 generic.go:356] "Generic (PLEG): container finished" podID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerID="d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea" exitCode=0
Sep 05 02:46:07.600131 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.600133    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"
Sep 05 02:46:07.600332 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.600152    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" event={"ID":"b7a753ea-9efe-4133-931a-e8edfd638d95","Type":"ContainerDied","Data":"d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea"}
Sep 05 02:46:07.600332 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.600209    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf" event={"ID":"b7a753ea-9efe-4133-931a-e8edfd638d95","Type":"ContainerDied","Data":"430c81a5704ee406b294637e7ffbed538ad435426f692c48995272d077aa50f4"}
Sep 05 02:46:07.600332 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.600239    2568 scope.go:117] "RemoveContainer" containerID="d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea"
Sep 05 02:46:07.608516 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.608490    2568 scope.go:117] "RemoveContainer" containerID="d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea"
Sep 05 02:46:07.608790 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:46:07.608773    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea\": container with ID starting with d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea not found: ID does not exist" containerID="d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea"
Sep 05 02:46:07.608849 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.608801    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea"} err="failed to get container status \"d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea\": rpc error: code = NotFound desc = could not find container \"d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea\": container with ID starting with d8db25a1f7e1241d8d5d56f890970a59919e819fcd6ef04745f9e741137d70ea not found: ID does not exist"
Sep 05 02:46:07.618940 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.618916    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"]
Sep 05 02:46:07.622953 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:07.622934    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcz9thf"]
Sep 05 02:46:09.031061 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:46:09.031024    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" path="/var/lib/kubelet/pods/b7a753ea-9efe-4133-931a-e8edfd638d95/volumes"
Sep 05 02:51:54.209341 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.209309    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"]
Sep 05 02:51:54.209780 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.209603    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main"
Sep 05 02:51:54.209780 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.209614    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main"
Sep 05 02:51:54.209780 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.209674    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a753ea-9efe-4133-931a-e8edfd638d95" containerName="main"
Sep 05 02:51:54.212418 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.212403    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.215378 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.215352    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 05 02:51:54.215518 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.215380    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 05 02:51:54.215518 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.215427    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:51:54.215518 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.215445    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 05 02:51:54.216549 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.216531    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 05 02:51:54.216549 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.216543    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tfjbj\""
Sep 05 02:51:54.229129 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.229110    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"]
Sep 05 02:51:54.272865 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.272838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-home\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.272974 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.272867    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.272974 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.272885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.272974 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.272909    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.273169 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.272981    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c08be327-8f78-4a74-90c7-36ec7da5c3e6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.273169 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.273014    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddfg\" (UniqueName: \"kubernetes.io/projected/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kube-api-access-pddfg\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.273169 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.273062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.273169 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.273111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-dshm\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373516 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373643 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-dshm\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373699 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373641    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-home\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373699 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373808 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373698    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373808 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373808 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373784    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c08be327-8f78-4a74-90c7-36ec7da5c3e6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373966 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373814    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddfg\" (UniqueName: \"kubernetes.io/projected/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kube-api-access-pddfg\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.373966 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373821    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.374063 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.373959    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-home\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.374063 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.374057    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.374208 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.374112    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.374427 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.374416    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c08be327-8f78-4a74-90c7-36ec7da5c3e6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.375983 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.375955    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-dshm\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.376167 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.376150    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.381335 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.381316    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddfg\" (UniqueName: \"kubernetes.io/projected/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kube-api-access-pddfg\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.522933 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.522855    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:51:54.798886 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.798859    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"]
Sep 05 02:51:54.801579 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:51:54.801550    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08be327_8f78_4a74_90c7_36ec7da5c3e6.slice/crio-896d1c0d05f7aa24d1cc946b69063233f1567e7602da672c6e05a96295e79288 WatchSource:0}: Error finding container 896d1c0d05f7aa24d1cc946b69063233f1567e7602da672c6e05a96295e79288: Status 404 returned error can't find the container with id 896d1c0d05f7aa24d1cc946b69063233f1567e7602da672c6e05a96295e79288
Sep 05 02:51:54.974162 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.974121    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" event={"ID":"c08be327-8f78-4a74-90c7-36ec7da5c3e6","Type":"ContainerStarted","Data":"d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257"}
Sep 05 02:51:54.974330 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:54.974187    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" event={"ID":"c08be327-8f78-4a74-90c7-36ec7da5c3e6","Type":"ContainerStarted","Data":"896d1c0d05f7aa24d1cc946b69063233f1567e7602da672c6e05a96295e79288"}
Sep 05 02:51:57.988004 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:57.987966    2568 generic.go:356] "Generic (PLEG): container finished" podID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerID="d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257" exitCode=0
Sep 05 02:51:57.988503 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:57.988045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" event={"ID":"c08be327-8f78-4a74-90c7-36ec7da5c3e6","Type":"ContainerDied","Data":"d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257"}
Sep 05 02:51:57.989225 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:57.989216    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:51:59.998339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:51:59.998307    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" event={"ID":"c08be327-8f78-4a74-90c7-36ec7da5c3e6","Type":"ContainerStarted","Data":"2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"}
Sep 05 02:52:00.020971 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:52:00.020917    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" podStartSLOduration=2.389260884 podStartE2EDuration="6.020901643s" totalImagesPullingTime="1.509540668s" totalInitContainerRuntime="2.122100091s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:52:00.020451277 +0000 UTC m=+2019.447567503" watchObservedRunningTime="2026-09-05 02:52:00.020901643 +0000 UTC m=+2019.448017867"
Sep 05 02:52:00.523133 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:52:00.523100    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:52:04.523359 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:52:04.523324    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:52:04.535834 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:52:04.535807    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:52:05.030476 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:52:05.030444    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:54:05.065802 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.065765    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"]
Sep 05 02:54:05.075428 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:54:05.075391    2568 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:54:05.075428 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:54:05.075428 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:54:05.075428 ip-10-0-140-68 kubenswrapper[2568]:  > containerID="2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376" cmd=["/bin/sleep","15"]
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.075461    2568 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" message="" err=<
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]:  >
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:54:05.075476    2568 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="main" containerID="cri-o://2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"
Sep 05 02:54:05.075639 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.075518    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="main" containerID="cri-o://2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376" gracePeriod=60
Sep 05 02:54:05.422752 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.422729    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:54:05.494497 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.494462    2568 generic.go:356] "Generic (PLEG): container finished" podID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerID="2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376" exitCode=0
Sep 05 02:54:05.494691 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.494513    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"
Sep 05 02:54:05.494691 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.494523    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" event={"ID":"c08be327-8f78-4a74-90c7-36ec7da5c3e6","Type":"ContainerDied","Data":"2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"}
Sep 05 02:54:05.494691 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.494577    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7" event={"ID":"c08be327-8f78-4a74-90c7-36ec7da5c3e6","Type":"ContainerDied","Data":"896d1c0d05f7aa24d1cc946b69063233f1567e7602da672c6e05a96295e79288"}
Sep 05 02:54:05.494691 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.494607    2568 scope.go:117] "RemoveContainer" containerID="2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"
Sep 05 02:54:05.503063 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.503046    2568 scope.go:117] "RemoveContainer" containerID="d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257"
Sep 05 02:54:05.510156 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.510136    2568 scope.go:117] "RemoveContainer" containerID="2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"
Sep 05 02:54:05.510415 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:54:05.510394    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376\": container with ID starting with 2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376 not found: ID does not exist" containerID="2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"
Sep 05 02:54:05.510465 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.510425    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376"} err="failed to get container status \"2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376\": rpc error: code = NotFound desc = could not find container \"2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376\": container with ID starting with 2f9a1f6972bf393c11ece63177e33687afb5515ec8e45fc7ab2d112c551dd376 not found: ID does not exist"
Sep 05 02:54:05.510465 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.510442    2568 scope.go:117] "RemoveContainer" containerID="d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257"
Sep 05 02:54:05.510664 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:54:05.510645    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257\": container with ID starting with d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257 not found: ID does not exist" containerID="d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257"
Sep 05 02:54:05.510699 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.510669    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257"} err="failed to get container status \"d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257\": rpc error: code = NotFound desc = could not find container \"d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257\": container with ID starting with d48934bbfd70a8331851da8e3e11def9bc42142806c02be88261da4990445257 not found: ID does not exist"
Sep 05 02:54:05.555736 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555708    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-home\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.555873 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555742    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tmp-dir\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.555873 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555758    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tls-certs\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.555873 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555776    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-model-cache\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.555873 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555806    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kserve-provision-location\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.556099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555890    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c08be327-8f78-4a74-90c7-36ec7da5c3e6-cabundle-cert\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.556099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.555957    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-dshm\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.556099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556005    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pddfg\" (UniqueName: \"kubernetes.io/projected/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kube-api-access-pddfg\") pod \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\" (UID: \"c08be327-8f78-4a74-90c7-36ec7da5c3e6\") "
Sep 05 02:54:05.556099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556041    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-home" (OuterVolumeSpecName: "home") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.556099 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556052    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.556362 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556126    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.556362 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556263    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.556362 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556280    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.556362 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556294    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.556362 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.556294    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08be327-8f78-4a74-90c7-36ec7da5c3e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:05.558299 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.558256    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kube-api-access-pddfg" (OuterVolumeSpecName: "kube-api-access-pddfg") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "kube-api-access-pddfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:05.558398 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.558296    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-dshm" (OuterVolumeSpecName: "dshm") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.558398 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.558334    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:05.575887 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.575862    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c08be327-8f78-4a74-90c7-36ec7da5c3e6" (UID: "c08be327-8f78-4a74-90c7-36ec7da5c3e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.657453 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.657421    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pddfg\" (UniqueName: \"kubernetes.io/projected/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kube-api-access-pddfg\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.657453 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.657448    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08be327-8f78-4a74-90c7-36ec7da5c3e6-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.657453 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.657461    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.657680 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.657470    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c08be327-8f78-4a74-90c7-36ec7da5c3e6-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.657680 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.657479    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08be327-8f78-4a74-90c7-36ec7da5c3e6-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.824049 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.824012    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"]
Sep 05 02:54:05.833509 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:05.833478    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-v6jx7"]
Sep 05 02:54:07.035702 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:54:07.035664    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" path="/var/lib/kubelet/pods/c08be327-8f78-4a74-90c7-36ec7da5c3e6/volumes"
Sep 05 02:55:14.914896 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.914857    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"]
Sep 05 02:55:14.915354 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.915139    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="main"
Sep 05 02:55:14.915354 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.915150    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="main"
Sep 05 02:55:14.915354 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.915166    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="storage-initializer"
Sep 05 02:55:14.915354 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.915172    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="storage-initializer"
Sep 05 02:55:14.915354 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.915213    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08be327-8f78-4a74-90c7-36ec7da5c3e6" containerName="main"
Sep 05 02:55:14.918147 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.918130    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.920811 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.920786    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:55:14.921021 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.920789    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 05 02:55:14.921133 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.921021    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 05 02:55:14.922063 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.922043    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 05 02:55:14.922261 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.922240    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 05 02:55:14.922339 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.922321    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9q65n\""
Sep 05 02:55:14.931015 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.930995    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"]
Sep 05 02:55:14.979092 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979264 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979098    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-dshm\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979264 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979134    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979264 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979264 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-home\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979399 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979288    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979399 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979318    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlvz\" (UniqueName: \"kubernetes.io/projected/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kube-api-access-cwlvz\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:14.979399 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:14.979338    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.080669 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080641    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-home\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.080815 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080676    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.080815 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080699    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlvz\" (UniqueName: \"kubernetes.io/projected/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kube-api-access-cwlvz\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.080815 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080727    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.080815 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.080815 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080800    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-dshm\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081124 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080822    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081124 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.080866    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081124 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.081064    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-home\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081248 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.081152    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081248 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.081230    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081569 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.081547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.081820 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.081583    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.083398 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.083374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.083487 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.083384    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-dshm\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.089291 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.089245    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlvz\" (UniqueName: \"kubernetes.io/projected/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kube-api-access-cwlvz\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.228611 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.228532    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:15.507128 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.507105    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"]
Sep 05 02:55:15.509273 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:55:15.509234    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8582ff5_28fd_4f1d_9aa0_334590c7ec6d.slice/crio-e978af1bac91785e995c922cfdbf9d272ebd140b6809176d8e99465c6795dbc4 WatchSource:0}: Error finding container e978af1bac91785e995c922cfdbf9d272ebd140b6809176d8e99465c6795dbc4: Status 404 returned error can't find the container with id e978af1bac91785e995c922cfdbf9d272ebd140b6809176d8e99465c6795dbc4
Sep 05 02:55:15.786265 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.786164    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" event={"ID":"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d","Type":"ContainerStarted","Data":"d22c19140ea36dfccccd20743e79b518b0ed90bca820e3f148949013eb5af40d"}
Sep 05 02:55:15.786265 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:15.786249    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" event={"ID":"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d","Type":"ContainerStarted","Data":"e978af1bac91785e995c922cfdbf9d272ebd140b6809176d8e99465c6795dbc4"}
Sep 05 02:55:17.797681 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:17.797647    2568 generic.go:356] "Generic (PLEG): container finished" podID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerID="d22c19140ea36dfccccd20743e79b518b0ed90bca820e3f148949013eb5af40d" exitCode=0
Sep 05 02:55:17.798112 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:17.797723    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" event={"ID":"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d","Type":"ContainerDied","Data":"d22c19140ea36dfccccd20743e79b518b0ed90bca820e3f148949013eb5af40d"}
Sep 05 02:55:18.804696 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:18.804657    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" event={"ID":"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d","Type":"ContainerStarted","Data":"f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e"}
Sep 05 02:55:18.827274 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:18.827225    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" podStartSLOduration=2.726753709 podStartE2EDuration="4.827209389s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.10045568s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:55:18.82602658 +0000 UTC m=+2218.253142828" watchObservedRunningTime="2026-09-05 02:55:18.827209389 +0000 UTC m=+2218.254325614"
Sep 05 02:55:19.229535 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:19.229505    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:25.229303 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:25.229266    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:25.241721 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:25.241692    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:25.843205 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:25.843179    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:52.629298 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.629208    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"]
Sep 05 02:55:52.638880 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:55:52.638847    2568 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:55:52.638880 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:55:52.638880 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:55:52.638880 ip-10-0-140-68 kubenswrapper[2568]:  > containerID="f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e" cmd=["/bin/sleep","15"]
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.638896    2568 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" message="" err=<
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]:  >
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:55:52.638908    2568 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="main" containerID="cri-o://f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e"
Sep 05 02:55:52.639191 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.638942    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="main" containerID="cri-o://f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e" gracePeriod=60
Sep 05 02:55:52.832500 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.832461    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Sep 05 02:55:52.946394 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.946357    2568 generic.go:356] "Generic (PLEG): container finished" podID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerID="f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e" exitCode=0
Sep 05 02:55:52.946541 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.946433    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" event={"ID":"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d","Type":"ContainerDied","Data":"f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e"}
Sep 05 02:55:52.989263 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:52.989235    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:53.075864 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.075830    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-dshm\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.075881    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kserve-provision-location\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.075934    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwlvz\" (UniqueName: \"kubernetes.io/projected/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kube-api-access-cwlvz\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.075981    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tls-certs\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076018 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076007    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-model-cache\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076254 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076036    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tmp-dir\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076254 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076102    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-cabundle-cert\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076254 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076125    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-home\") pod \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\" (UID: \"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d\") "
Sep 05 02:55:53.076400 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076320    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.076400 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076386    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.076552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076531    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-home" (OuterVolumeSpecName: "home") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.076656 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.076548    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:53.078409 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.078378    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kube-api-access-cwlvz" (OuterVolumeSpecName: "kube-api-access-cwlvz") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "kube-api-access-cwlvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:55:53.078529 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.078415    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-dshm" (OuterVolumeSpecName: "dshm") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.078529 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.078421    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:53.095388 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.095367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" (UID: "e8582ff5-28fd-4f1d-9aa0-334590c7ec6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.177098 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177019    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177098 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177043    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177098 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177051    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177098 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177058    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177098 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177066    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177098 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177095    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177364 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177109    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.177364 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.177117    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlvz\" (UniqueName: \"kubernetes.io/projected/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d-kube-api-access-cwlvz\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.952292 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.952249    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7" event={"ID":"e8582ff5-28fd-4f1d-9aa0-334590c7ec6d","Type":"ContainerDied","Data":"e978af1bac91785e995c922cfdbf9d272ebd140b6809176d8e99465c6795dbc4"}
Sep 05 02:55:53.952730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.952297    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"
Sep 05 02:55:53.952730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.952322    2568 scope.go:117] "RemoveContainer" containerID="f63b9626f683436aa67ce4b05dab6a9d17a8064d46c2f8aa15b65209b991980e"
Sep 05 02:55:53.961455 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.961439    2568 scope.go:117] "RemoveContainer" containerID="d22c19140ea36dfccccd20743e79b518b0ed90bca820e3f148949013eb5af40d"
Sep 05 02:55:53.977566 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.977540    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"]
Sep 05 02:55:53.987148 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:53.987125    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-jqqv7"]
Sep 05 02:55:55.030490 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:55:55.030451    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" path="/var/lib/kubelet/pods/e8582ff5-28fd-4f1d-9aa0-334590c7ec6d/volumes"
Sep 05 02:56:51.839162 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.839125    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"]
Sep 05 02:56:51.839606 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.839411    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="main"
Sep 05 02:56:51.839606 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.839422    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="main"
Sep 05 02:56:51.839606 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.839435    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="storage-initializer"
Sep 05 02:56:51.839606 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.839441    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="storage-initializer"
Sep 05 02:56:51.839606 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.839493    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8582ff5-28fd-4f1d-9aa0-334590c7ec6d" containerName="main"
Sep 05 02:56:51.842357 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.842339    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.848992 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.848888    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 05 02:56:51.848992 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.848978    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 05 02:56:51.849664 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.849646    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-86nxw\""
Sep 05 02:56:51.849766 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.849647    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 05 02:56:51.850123 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.850106    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:56:51.850270 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.850113    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 05 02:56:51.878479 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.878449    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"]
Sep 05 02:56:51.997439 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997406    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758fb39d-5429-4bb0-b43f-1091c3495670-tls-certs\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997443    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997468    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-home\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997552 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997528    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-model-cache\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997661 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-tmp-dir\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997661 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/758fb39d-5429-4bb0-b43f-1091c3495670-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997661 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997606    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdmm\" (UniqueName: \"kubernetes.io/projected/758fb39d-5429-4bb0-b43f-1091c3495670-kube-api-access-bpdmm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:51.997756 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:51.997698    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-dshm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.098682 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/758fb39d-5429-4bb0-b43f-1091c3495670-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.098682 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdmm\" (UniqueName: \"kubernetes.io/projected/758fb39d-5429-4bb0-b43f-1091c3495670-kube-api-access-bpdmm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.098861 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098717    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-dshm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.098861 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098761    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758fb39d-5429-4bb0-b43f-1091c3495670-tls-certs\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.098861 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098796    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.098861 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098827    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-home\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099062 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-model-cache\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099062 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.098909    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-tmp-dir\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099285 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.099262    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-tmp-dir\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099480 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.099454    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-home\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099480 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.099469    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/758fb39d-5429-4bb0-b43f-1091c3495670-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099710 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.099685    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-model-cache\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.099831 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.099694    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.101402 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.101381    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-dshm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.101790 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.101771    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758fb39d-5429-4bb0-b43f-1091c3495670-tls-certs\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.111433 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.111413    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdmm\" (UniqueName: \"kubernetes.io/projected/758fb39d-5429-4bb0-b43f-1091c3495670-kube-api-access-bpdmm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-njtph\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.152791 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.152767    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:56:52.437234 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:52.437201    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"]
Sep 05 02:56:52.439679 ip-10-0-140-68 kubenswrapper[2568]: W0905 02:56:52.439652    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758fb39d_5429_4bb0_b43f_1091c3495670.slice/crio-1d2aea60e6c7030a941672588d0349371f3fd0e4e77320f608f85e1288dd367b WatchSource:0}: Error finding container 1d2aea60e6c7030a941672588d0349371f3fd0e4e77320f608f85e1288dd367b: Status 404 returned error can't find the container with id 1d2aea60e6c7030a941672588d0349371f3fd0e4e77320f608f85e1288dd367b
Sep 05 02:56:53.192245 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:53.192204    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" event={"ID":"758fb39d-5429-4bb0-b43f-1091c3495670","Type":"ContainerStarted","Data":"d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a"}
Sep 05 02:56:53.192628 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:53.192273    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" event={"ID":"758fb39d-5429-4bb0-b43f-1091c3495670","Type":"ContainerStarted","Data":"1d2aea60e6c7030a941672588d0349371f3fd0e4e77320f608f85e1288dd367b"}
Sep 05 02:56:55.203122 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:55.203067    2568 generic.go:356] "Generic (PLEG): container finished" podID="758fb39d-5429-4bb0-b43f-1091c3495670" containerID="d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a" exitCode=0
Sep 05 02:56:55.203122 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:55.203126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" event={"ID":"758fb39d-5429-4bb0-b43f-1091c3495670","Type":"ContainerDied","Data":"d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a"}
Sep 05 02:56:56.209741 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:56.209701    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" event={"ID":"758fb39d-5429-4bb0-b43f-1091c3495670","Type":"ContainerStarted","Data":"d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"}
Sep 05 02:56:56.231147 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:56.231101    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" podStartSLOduration=3.125781865 podStartE2EDuration="5.231068279s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.105286414s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:56.229329828 +0000 UTC m=+2315.656446056" watchObservedRunningTime="2026-09-05 02:56:56.231068279 +0000 UTC m=+2315.658184503"
Sep 05 02:56:57.153656 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:56:57.153607    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:57:02.153730 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:02.153689    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:57:02.166533 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:02.166502    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:57:02.244667 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:02.244636    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:57:54.041723 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.041681    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"]
Sep 05 02:57:54.050997 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:57:54.050960    2568 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:57:54.050997 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:57:54.050997 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:57:54.050997 ip-10-0-140-68 kubenswrapper[2568]:  > containerID="d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94" cmd=["/bin/sleep","15"]
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.051011    2568 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" message="" err=<
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]:  >
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:57:54.051025    2568 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]:         , stdout: , stderr: , exit code -1
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="main" containerID="cri-o://d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"
Sep 05 02:57:54.051281 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.051101    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="main" containerID="cri-o://d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94" gracePeriod=60
Sep 05 02:57:54.233622 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.233583    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 05 02:57:54.393145 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.393124    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:57:54.452026 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.451994    2568 generic.go:356] "Generic (PLEG): container finished" podID="758fb39d-5429-4bb0-b43f-1091c3495670" containerID="d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94" exitCode=0
Sep 05 02:57:54.452195 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.452056    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"
Sep 05 02:57:54.452195 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.452068    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" event={"ID":"758fb39d-5429-4bb0-b43f-1091c3495670","Type":"ContainerDied","Data":"d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"}
Sep 05 02:57:54.452195 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.452139    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph" event={"ID":"758fb39d-5429-4bb0-b43f-1091c3495670","Type":"ContainerDied","Data":"1d2aea60e6c7030a941672588d0349371f3fd0e4e77320f608f85e1288dd367b"}
Sep 05 02:57:54.452195 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.452161    2568 scope.go:117] "RemoveContainer" containerID="d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"
Sep 05 02:57:54.460402 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.460386    2568 scope.go:117] "RemoveContainer" containerID="d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a"
Sep 05 02:57:54.467633 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.467611    2568 scope.go:117] "RemoveContainer" containerID="d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"
Sep 05 02:57:54.467885 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:57:54.467868    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94\": container with ID starting with d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94 not found: ID does not exist" containerID="d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"
Sep 05 02:57:54.467937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.467894    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94"} err="failed to get container status \"d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94\": rpc error: code = NotFound desc = could not find container \"d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94\": container with ID starting with d0ce6366fbba35708a88b3916f5a05291e3dba01f27f1d539474316ec3665c94 not found: ID does not exist"
Sep 05 02:57:54.467937 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.467912    2568 scope.go:117] "RemoveContainer" containerID="d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a"
Sep 05 02:57:54.468175 ip-10-0-140-68 kubenswrapper[2568]: E0905 02:57:54.468158    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a\": container with ID starting with d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a not found: ID does not exist" containerID="d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a"
Sep 05 02:57:54.468225 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.468183    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a"} err="failed to get container status \"d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a\": rpc error: code = NotFound desc = could not find container \"d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a\": container with ID starting with d4e9183e4c6ff10054b9ea01548e4db4217390cb61c393a915481c415339ed3a not found: ID does not exist"
Sep 05 02:57:54.469440 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469427    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758fb39d-5429-4bb0-b43f-1091c3495670-tls-certs\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469482 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469457    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-kserve-provision-location\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469519 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469502    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpdmm\" (UniqueName: \"kubernetes.io/projected/758fb39d-5429-4bb0-b43f-1091c3495670-kube-api-access-bpdmm\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469563 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469549    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-home\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469603 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469578    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-dshm\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469639 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469612    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/758fb39d-5429-4bb0-b43f-1091c3495670-cabundle-cert\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469688 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469644    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-model-cache\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.469688 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469673    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-tmp-dir\") pod \"758fb39d-5429-4bb0-b43f-1091c3495670\" (UID: \"758fb39d-5429-4bb0-b43f-1091c3495670\") "
Sep 05 02:57:54.470471 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469781    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-home" (OuterVolumeSpecName: "home") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.470471 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469949    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.470471 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.469989    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.470471 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.470000    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758fb39d-5429-4bb0-b43f-1091c3495670-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:54.470471 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.470187    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-model-cache" (OuterVolumeSpecName: "model-cache") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.471706 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.471679    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758fb39d-5429-4bb0-b43f-1091c3495670-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:57:54.471850 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.471830    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758fb39d-5429-4bb0-b43f-1091c3495670-kube-api-access-bpdmm" (OuterVolumeSpecName: "kube-api-access-bpdmm") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "kube-api-access-bpdmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:57:54.471850 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.471849    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-dshm" (OuterVolumeSpecName: "dshm") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.489474 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.489449    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "758fb39d-5429-4bb0-b43f-1091c3495670" (UID: "758fb39d-5429-4bb0-b43f-1091c3495670"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.570309 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570252    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.570309 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570275    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/758fb39d-5429-4bb0-b43f-1091c3495670-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.570309 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570288    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.570309 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570302    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.570486 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570313    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758fb39d-5429-4bb0-b43f-1091c3495670-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.570486 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570326    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758fb39d-5429-4bb0-b43f-1091c3495670-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.570486 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.570338    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdmm\" (UniqueName: \"kubernetes.io/projected/758fb39d-5429-4bb0-b43f-1091c3495670-kube-api-access-bpdmm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.772281 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.772251    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"]
Sep 05 02:57:54.775799 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:54.775778    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-njtph"]
Sep 05 02:57:55.030177 ip-10-0-140-68 kubenswrapper[2568]: I0905 02:57:55.030148    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" path="/var/lib/kubelet/pods/758fb39d-5429-4bb0-b43f-1091c3495670/volumes"
Sep 05 03:01:50.967295 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.967217    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"]
Sep 05 03:01:50.967745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.967515    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="main"
Sep 05 03:01:50.967745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.967531    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="main"
Sep 05 03:01:50.967745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.967557    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="storage-initializer"
Sep 05 03:01:50.967745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.967563    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="storage-initializer"
Sep 05 03:01:50.967745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.967606    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="758fb39d-5429-4bb0-b43f-1091c3495670" containerName="main"
Sep 05 03:01:50.970359 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.970341    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:50.973087 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.973047    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 05 03:01:50.974478 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.974458    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-swzwf\""
Sep 05 03:01:50.974580 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.974525    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 05 03:01:50.974580 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.974571    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-m6spr\""
Sep 05 03:01:50.974688 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.974611    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 05 03:01:50.983990 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:50.983963    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"]
Sep 05 03:01:51.077141 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.077063    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.077141 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.077147    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krpw\" (UniqueName: \"kubernetes.io/projected/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-kube-api-access-6krpw\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.178128 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.178094    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.178292 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.178145    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krpw\" (UniqueName: \"kubernetes.io/projected/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-kube-api-access-6krpw\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.180755 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.180730    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.187515 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.187489    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krpw\" (UniqueName: \"kubernetes.io/projected/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-kube-api-access-6krpw\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.281441 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.281348    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:51.553428 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:51.553353    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"]
Sep 05 03:01:51.556182 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:01:51.556149    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7697eb4_2788_454e_8b89_f1ca2cacc5d8.slice/crio-56e25bfaa5e563751236dadb97c46c29ebc268cace0142f3ae7d4f9cae3613b9 WatchSource:0}: Error finding container 56e25bfaa5e563751236dadb97c46c29ebc268cace0142f3ae7d4f9cae3613b9: Status 404 returned error can't find the container with id 56e25bfaa5e563751236dadb97c46c29ebc268cace0142f3ae7d4f9cae3613b9
Sep 05 03:01:52.384493 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:52.384453    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" event={"ID":"e7697eb4-2788-454e-8b89-f1ca2cacc5d8","Type":"ContainerStarted","Data":"d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5"}
Sep 05 03:01:52.384850 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:52.384539    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" event={"ID":"e7697eb4-2788-454e-8b89-f1ca2cacc5d8","Type":"ContainerStarted","Data":"56e25bfaa5e563751236dadb97c46c29ebc268cace0142f3ae7d4f9cae3613b9"}
Sep 05 03:01:52.384850 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:52.384580    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:01:52.404212 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:01:52.404166    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" podStartSLOduration=2.404151209 podStartE2EDuration="2.404151209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:01:52.402239449 +0000 UTC m=+2611.829355675" watchObservedRunningTime="2026-09-05 03:01:52.404151209 +0000 UTC m=+2611.831267433"
Sep 05 03:02:23.390771 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:23.390738    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:02:25.297997 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:25.297965    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"]
Sep 05 03:02:33.390674 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:33.390635    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:02:40.318192 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:40.318133    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main" containerID="cri-o://d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5" gracePeriod=45
Sep 05 03:02:43.391335 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:43.391296    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:02:53.390568 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:53.390526    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:02:53.391108 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:02:53.390625    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:03:03.391345 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:03.391306    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:03:10.553191 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.553163    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:03:10.677428 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.677398    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6krpw\" (UniqueName: \"kubernetes.io/projected/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-kube-api-access-6krpw\") pod \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") "
Sep 05 03:03:10.677584 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.677458    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-tls-certs\") pod \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\" (UID: \"e7697eb4-2788-454e-8b89-f1ca2cacc5d8\") "
Sep 05 03:03:10.679847 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.679821    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-kube-api-access-6krpw" (OuterVolumeSpecName: "kube-api-access-6krpw") pod "e7697eb4-2788-454e-8b89-f1ca2cacc5d8" (UID: "e7697eb4-2788-454e-8b89-f1ca2cacc5d8"). InnerVolumeSpecName "kube-api-access-6krpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:03:10.679964 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.679821    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7697eb4-2788-454e-8b89-f1ca2cacc5d8" (UID: "e7697eb4-2788-454e-8b89-f1ca2cacc5d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:03:10.695728 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.695698    2568 generic.go:356] "Generic (PLEG): container finished" podID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerID="d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5" exitCode=0
Sep 05 03:03:10.695848 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.695747    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"
Sep 05 03:03:10.695848 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.695764    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" event={"ID":"e7697eb4-2788-454e-8b89-f1ca2cacc5d8","Type":"ContainerDied","Data":"d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5"}
Sep 05 03:03:10.695848 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.695815    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9" event={"ID":"e7697eb4-2788-454e-8b89-f1ca2cacc5d8","Type":"ContainerDied","Data":"56e25bfaa5e563751236dadb97c46c29ebc268cace0142f3ae7d4f9cae3613b9"}
Sep 05 03:03:10.695848 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.695838    2568 scope.go:117] "RemoveContainer" containerID="d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5"
Sep 05 03:03:10.704519 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.704504    2568 scope.go:117] "RemoveContainer" containerID="d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5"
Sep 05 03:03:10.704784 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:03:10.704767    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5\": container with ID starting with d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5 not found: ID does not exist" containerID="d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5"
Sep 05 03:03:10.704831 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.704792    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5"} err="failed to get container status \"d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5\": rpc error: code = NotFound desc = could not find container \"d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5\": container with ID starting with d5a685a864e04d59cd3dae04b83c4b851aded5be4a01408d78d34a69f5d6a6e5 not found: ID does not exist"
Sep 05 03:03:10.715759 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.715735    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"]
Sep 05 03:03:10.720028 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.720007    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schendjt9"]
Sep 05 03:03:10.778136 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.778108    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:03:10.778136 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:10.778133    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krpw\" (UniqueName: \"kubernetes.io/projected/e7697eb4-2788-454e-8b89-f1ca2cacc5d8-kube-api-access-6krpw\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:03:11.030226 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:03:11.030144    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" path="/var/lib/kubelet/pods/e7697eb4-2788-454e-8b89-f1ca2cacc5d8/volumes"
Sep 05 03:05:10.630593 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.630545    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"]
Sep 05 03:05:10.631213 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.630845    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main"
Sep 05 03:05:10.631213 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.630856    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main"
Sep 05 03:05:10.631213 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.630909    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7697eb4-2788-454e-8b89-f1ca2cacc5d8" containerName="main"
Sep 05 03:05:10.633736 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.633717    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.637460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.637427    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-bz4ww\""
Sep 05 03:05:10.637603 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.637579    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 05 03:05:10.637730 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.637693    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 05 03:05:10.637838 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.637822    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-zb25w\""
Sep 05 03:05:10.637900 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.637881    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 05 03:05:10.647244 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.647214    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"]
Sep 05 03:05:10.670003 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.669974    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08bbfc35-3d92-41e1-b826-bb4114506506-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.670129 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.670010    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksd9r\" (UniqueName: \"kubernetes.io/projected/08bbfc35-3d92-41e1-b826-bb4114506506-kube-api-access-ksd9r\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.770501 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.770468    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08bbfc35-3d92-41e1-b826-bb4114506506-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.770630 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.770509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksd9r\" (UniqueName: \"kubernetes.io/projected/08bbfc35-3d92-41e1-b826-bb4114506506-kube-api-access-ksd9r\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.773091 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.773056    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08bbfc35-3d92-41e1-b826-bb4114506506-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.779336 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.779312    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksd9r\" (UniqueName: \"kubernetes.io/projected/08bbfc35-3d92-41e1-b826-bb4114506506-kube-api-access-ksd9r\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:10.946507 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:10.946425    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:11.223257 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:11.223232    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"]
Sep 05 03:05:11.225601 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:05:11.225569    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bbfc35_3d92_41e1_b826_bb4114506506.slice/crio-cd88a5b6b52e56e11531d044ba426581c00761cb5371ab49d62f9a700065dc28 WatchSource:0}: Error finding container cd88a5b6b52e56e11531d044ba426581c00761cb5371ab49d62f9a700065dc28: Status 404 returned error can't find the container with id cd88a5b6b52e56e11531d044ba426581c00761cb5371ab49d62f9a700065dc28
Sep 05 03:05:12.181958 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:12.181924    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" event={"ID":"08bbfc35-3d92-41e1-b826-bb4114506506","Type":"ContainerStarted","Data":"fa54bf372de6c9c8d59efd20d7aac41450f879443f9939e68876b2c520d38f07"}
Sep 05 03:05:12.182351 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:12.181975    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" event={"ID":"08bbfc35-3d92-41e1-b826-bb4114506506","Type":"ContainerStarted","Data":"cd88a5b6b52e56e11531d044ba426581c00761cb5371ab49d62f9a700065dc28"}
Sep 05 03:05:12.182351 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:12.182066    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:05:12.200166 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:12.200123    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podStartSLOduration=2.200109084 podStartE2EDuration="2.200109084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:05:12.199458731 +0000 UTC m=+2811.626574957" watchObservedRunningTime="2026-09-05 03:05:12.200109084 +0000 UTC m=+2811.627225306"
Sep 05 03:05:43.189097 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:05:43.189043    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:06:01.244745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.244710    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"]
Sep 05 03:06:01.248015 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.248000    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.251783 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.251758    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 05 03:06:01.251970 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.251758    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 05 03:06:01.252045 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.252009    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 05 03:06:01.252168 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.252144    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:06:01.252237 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.252166    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 05 03:06:01.254016 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.253999    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-hvhmc\""
Sep 05 03:06:01.272974 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.272951    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"]
Sep 05 03:06:01.328582 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328557    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328687 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328595    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328687 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328627    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328687 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328654    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e38fc-8084-46f5-a289-6b78d4a3b702-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328811 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328811 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328759    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbxh\" (UniqueName: \"kubernetes.io/projected/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kube-api-access-ckbxh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328811 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328786    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.328931 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.328828    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429260 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429222    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbxh\" (UniqueName: \"kubernetes.io/projected/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kube-api-access-ckbxh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429271    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429307    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429332    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429399    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e38fc-8084-46f5-a289-6b78d4a3b702-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429735 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429435    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429790 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429766    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429901 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429878    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429953 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.429953 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.429944    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.430155 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.430131    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e38fc-8084-46f5-a289-6b78d4a3b702-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.431875 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.431853    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.432032 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.432012    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.437283 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.437252    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbxh\" (UniqueName: \"kubernetes.io/projected/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kube-api-access-ckbxh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.557957 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.557888    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:01.846789 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:01.846696    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"]
Sep 05 03:06:01.850720 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:06:01.850681    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2e38fc_8084_46f5_a289_6b78d4a3b702.slice/crio-448778cf7f8b652539537cc4983a33a2f428ea40171baedcccd8c0d9f9ada72f WatchSource:0}: Error finding container 448778cf7f8b652539537cc4983a33a2f428ea40171baedcccd8c0d9f9ada72f: Status 404 returned error can't find the container with id 448778cf7f8b652539537cc4983a33a2f428ea40171baedcccd8c0d9f9ada72f
Sep 05 03:06:02.386314 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:02.386276    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" event={"ID":"0e2e38fc-8084-46f5-a289-6b78d4a3b702","Type":"ContainerStarted","Data":"a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf"}
Sep 05 03:06:02.386737 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:02.386331    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" event={"ID":"0e2e38fc-8084-46f5-a289-6b78d4a3b702","Type":"ContainerStarted","Data":"448778cf7f8b652539537cc4983a33a2f428ea40171baedcccd8c0d9f9ada72f"}
Sep 05 03:06:04.400971 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:04.400934    2568 generic.go:356] "Generic (PLEG): container finished" podID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerID="a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf" exitCode=0
Sep 05 03:06:04.401350 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:04.401009    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" event={"ID":"0e2e38fc-8084-46f5-a289-6b78d4a3b702","Type":"ContainerDied","Data":"a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf"}
Sep 05 03:06:05.407580 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:05.407539    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" event={"ID":"0e2e38fc-8084-46f5-a289-6b78d4a3b702","Type":"ContainerStarted","Data":"9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9"}
Sep 05 03:06:05.433901 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:05.433849    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podStartSLOduration=2.349273182 podStartE2EDuration="4.43383299s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.084559808s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:06:05.431589946 +0000 UTC m=+2864.858706181" watchObservedRunningTime="2026-09-05 03:06:05.43383299 +0000 UTC m=+2864.860949214"
Sep 05 03:06:05.558558 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:05.558520    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:11.558248 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:11.558211    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:06:11.559485 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:11.559454    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:06:18.422585 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:18.422544    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"]
Sep 05 03:06:21.558382 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:21.558341    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:06:23.189242 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:23.189200    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:06:31.559012 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:31.558963    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:06:33.188676 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:33.188637    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:06:33.444047 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:33.443937    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main" containerID="cri-o://fa54bf372de6c9c8d59efd20d7aac41450f879443f9939e68876b2c520d38f07" gracePeriod=45
Sep 05 03:06:41.559186 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:41.559128    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:06:43.189470 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:43.189426    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:06:43.189925 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:43.189570    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:06:51.558379 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:51.558316    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:06:53.189213 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:06:53.189171    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:07:01.558739 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:01.558686    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:07:03.188693 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.188649    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:07:03.672913 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.672878    2568 generic.go:356] "Generic (PLEG): container finished" podID="08bbfc35-3d92-41e1-b826-bb4114506506" containerID="fa54bf372de6c9c8d59efd20d7aac41450f879443f9939e68876b2c520d38f07" exitCode=0
Sep 05 03:07:03.673115 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.672956    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" event={"ID":"08bbfc35-3d92-41e1-b826-bb4114506506","Type":"ContainerDied","Data":"fa54bf372de6c9c8d59efd20d7aac41450f879443f9939e68876b2c520d38f07"}
Sep 05 03:07:03.705047 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.704979    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:07:03.857503 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.857468    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08bbfc35-3d92-41e1-b826-bb4114506506-tls-certs\") pod \"08bbfc35-3d92-41e1-b826-bb4114506506\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") "
Sep 05 03:07:03.857671 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.857512    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksd9r\" (UniqueName: \"kubernetes.io/projected/08bbfc35-3d92-41e1-b826-bb4114506506-kube-api-access-ksd9r\") pod \"08bbfc35-3d92-41e1-b826-bb4114506506\" (UID: \"08bbfc35-3d92-41e1-b826-bb4114506506\") "
Sep 05 03:07:03.859877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.859838    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bbfc35-3d92-41e1-b826-bb4114506506-kube-api-access-ksd9r" (OuterVolumeSpecName: "kube-api-access-ksd9r") pod "08bbfc35-3d92-41e1-b826-bb4114506506" (UID: "08bbfc35-3d92-41e1-b826-bb4114506506"). InnerVolumeSpecName "kube-api-access-ksd9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:07:03.860010 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.859929    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08bbfc35-3d92-41e1-b826-bb4114506506-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08bbfc35-3d92-41e1-b826-bb4114506506" (UID: "08bbfc35-3d92-41e1-b826-bb4114506506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:07:03.958481 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.958375    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksd9r\" (UniqueName: \"kubernetes.io/projected/08bbfc35-3d92-41e1-b826-bb4114506506-kube-api-access-ksd9r\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:07:03.958481 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:03.958419    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08bbfc35-3d92-41e1-b826-bb4114506506-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:07:04.679270 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:04.679238    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"
Sep 05 03:07:04.679270 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:04.679252    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg" event={"ID":"08bbfc35-3d92-41e1-b826-bb4114506506","Type":"ContainerDied","Data":"cd88a5b6b52e56e11531d044ba426581c00761cb5371ab49d62f9a700065dc28"}
Sep 05 03:07:04.679814 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:04.679311    2568 scope.go:117] "RemoveContainer" containerID="fa54bf372de6c9c8d59efd20d7aac41450f879443f9939e68876b2c520d38f07"
Sep 05 03:07:04.702639 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:04.702598    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"]
Sep 05 03:07:04.706979 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:04.706945    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schehbcbg"]
Sep 05 03:07:05.031332 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:05.031247    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" path="/var/lib/kubelet/pods/08bbfc35-3d92-41e1-b826-bb4114506506/volumes"
Sep 05 03:07:11.559124 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:11.559035    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:07:21.559004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:21.558908    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:07:31.558917 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:31.558872    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:07:41.558867 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:41.558824    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:07:51.559261 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:07:51.559178    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:08:01.568557 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:08:01.568527    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:08:01.581106 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:08:01.581053    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:09:10.227413 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:10.227379    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"]
Sep 05 03:09:25.251086 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:25.251037    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" containerID="cri-o://9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9" gracePeriod=45
Sep 05 03:09:25.570183 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:25.570064    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 05 03:09:25.925826 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:25.925800    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:09:26.110588 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110485    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-dshm\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110588 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110563    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tls-certs\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110834 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110602    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tmp-dir\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110834 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110619    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e38fc-8084-46f5-a289-6b78d4a3b702-cabundle-cert\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110834 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110645    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-model-cache\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110834 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110669    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-home\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110834 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110712    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kserve-provision-location\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.110834 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110760    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckbxh\" (UniqueName: \"kubernetes.io/projected/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kube-api-access-ckbxh\") pod \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\" (UID: \"0e2e38fc-8084-46f5-a289-6b78d4a3b702\") "
Sep 05 03:09:26.111178 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.110941    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:09:26.111178 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.111030    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2e38fc-8084-46f5-a289-6b78d4a3b702-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:09:26.111647 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.111610    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-home" (OuterVolumeSpecName: "home") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:09:26.113014 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.112993    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-dshm" (OuterVolumeSpecName: "dshm") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:09:26.113172 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.113149    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:09:26.113662 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.113638    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kube-api-access-ckbxh" (OuterVolumeSpecName: "kube-api-access-ckbxh") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "kube-api-access-ckbxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:09:26.122841 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.122816    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:09:26.132294 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.132268    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e2e38fc-8084-46f5-a289-6b78d4a3b702" (UID: "0e2e38fc-8084-46f5-a289-6b78d4a3b702"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:09:26.211959 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211924    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckbxh\" (UniqueName: \"kubernetes.io/projected/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kube-api-access-ckbxh\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.211959 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211954    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.211959 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211964    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.212199 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211973    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.212199 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211982    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e38fc-8084-46f5-a289-6b78d4a3b702-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.212199 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211989    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.212199 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.211996    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.212199 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.212004    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2e38fc-8084-46f5-a289-6b78d4a3b702-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:09:26.291231 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.291197    2568 generic.go:356] "Generic (PLEG): container finished" podID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerID="9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9" exitCode=0
Sep 05 03:09:26.291679 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.291249    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" event={"ID":"0e2e38fc-8084-46f5-a289-6b78d4a3b702","Type":"ContainerDied","Data":"9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9"}
Sep 05 03:09:26.291679 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.291270    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"
Sep 05 03:09:26.291679 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.291293    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w" event={"ID":"0e2e38fc-8084-46f5-a289-6b78d4a3b702","Type":"ContainerDied","Data":"448778cf7f8b652539537cc4983a33a2f428ea40171baedcccd8c0d9f9ada72f"}
Sep 05 03:09:26.291679 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.291315    2568 scope.go:117] "RemoveContainer" containerID="9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9"
Sep 05 03:09:26.305710 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.305684    2568 scope.go:117] "RemoveContainer" containerID="a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf"
Sep 05 03:09:26.313869 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.313843    2568 scope.go:117] "RemoveContainer" containerID="9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9"
Sep 05 03:09:26.314164 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:09:26.314143    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9\": container with ID starting with 9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9 not found: ID does not exist" containerID="9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9"
Sep 05 03:09:26.314215 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.314176    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9"} err="failed to get container status \"9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9\": rpc error: code = NotFound desc = could not find container \"9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9\": container with ID starting with 9c71885662c5b8ce33b34779c79ece3c9adca989a4c05064a1a7fb074b12a1a9 not found: ID does not exist"
Sep 05 03:09:26.314215 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.314196    2568 scope.go:117] "RemoveContainer" containerID="a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf"
Sep 05 03:09:26.314469 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:09:26.314449    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf\": container with ID starting with a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf not found: ID does not exist" containerID="a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf"
Sep 05 03:09:26.314509 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.314477    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf"} err="failed to get container status \"a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf\": rpc error: code = NotFound desc = could not find container \"a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf\": container with ID starting with a408e33a8e15bd5f8625355d17ef2f813967d20080b8a47bedd535f4fe4dd0bf not found: ID does not exist"
Sep 05 03:09:26.318013 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.317990    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"]
Sep 05 03:09:26.323727 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:26.323707    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-777f747bd69bb2w"]
Sep 05 03:09:27.030481 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:09:27.030449    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" path="/var/lib/kubelet/pods/0e2e38fc-8084-46f5-a289-6b78d4a3b702/volumes"
Sep 05 03:14:22.569694 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.569663    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"]
Sep 05 03:14:22.570209 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570043    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="storage-initializer"
Sep 05 03:14:22.570209 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570059    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="storage-initializer"
Sep 05 03:14:22.570209 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570110    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main"
Sep 05 03:14:22.570209 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570120    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main"
Sep 05 03:14:22.570209 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570132    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main"
Sep 05 03:14:22.570209 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570140    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main"
Sep 05 03:14:22.570518 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570215    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bbfc35-3d92-41e1-b826-bb4114506506" containerName="main"
Sep 05 03:14:22.570518 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.570226    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2e38fc-8084-46f5-a289-6b78d4a3b702" containerName="main"
Sep 05 03:14:22.573067 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.573046    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.577008 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.576987    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 05 03:14:22.577132 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.577110    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 05 03:14:22.577220 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.577044    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qgpqf\""
Sep 05 03:14:22.577427 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.577407    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 05 03:14:22.577542 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.577429    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-j4knx\""
Sep 05 03:14:22.581722 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.581702    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"]
Sep 05 03:14:22.647475 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.647451    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac938c48-2041-4379-abe1-7e56c000d462-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.647575 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.647486    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4hk\" (UniqueName: \"kubernetes.io/projected/ac938c48-2041-4379-abe1-7e56c000d462-kube-api-access-9k4hk\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.748851 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.748818    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac938c48-2041-4379-abe1-7e56c000d462-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.748980 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.748868    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4hk\" (UniqueName: \"kubernetes.io/projected/ac938c48-2041-4379-abe1-7e56c000d462-kube-api-access-9k4hk\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.751451 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.751426    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac938c48-2041-4379-abe1-7e56c000d462-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.757330 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.757300    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4hk\" (UniqueName: \"kubernetes.io/projected/ac938c48-2041-4379-abe1-7e56c000d462-kube-api-access-9k4hk\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:22.885119 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:22.885098    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:23.157378 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:23.157302    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"]
Sep 05 03:14:23.160295 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:14:23.160268    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac938c48_2041_4379_abe1_7e56c000d462.slice/crio-37140585087a5680540c7625c22ab2caa958c93b10f0d3e50d133938d9d11f2e WatchSource:0}: Error finding container 37140585087a5680540c7625c22ab2caa958c93b10f0d3e50d133938d9d11f2e: Status 404 returned error can't find the container with id 37140585087a5680540c7625c22ab2caa958c93b10f0d3e50d133938d9d11f2e
Sep 05 03:14:23.514192 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:23.514093    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" event={"ID":"ac938c48-2041-4379-abe1-7e56c000d462","Type":"ContainerStarted","Data":"23bb6acd77f0eba22d8d2897439b06fa041100063858afdc0f83deef3c2f6a0a"}
Sep 05 03:14:23.514192 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:23.514159    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" event={"ID":"ac938c48-2041-4379-abe1-7e56c000d462","Type":"ContainerStarted","Data":"37140585087a5680540c7625c22ab2caa958c93b10f0d3e50d133938d9d11f2e"}
Sep 05 03:14:23.514192 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:23.514187    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:14:23.531410 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:23.531363    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" podStartSLOduration=1.53134888 podStartE2EDuration="1.53134888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:14:23.529760297 +0000 UTC m=+3362.956876523" watchObservedRunningTime="2026-09-05 03:14:23.53134888 +0000 UTC m=+3362.958465107"
Sep 05 03:14:25.243980 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:25.243942    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"]
Sep 05 03:14:40.543200 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:40.543132    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" podUID="ac938c48-2041-4379-abe1-7e56c000d462" containerName="main" containerID="cri-o://23bb6acd77f0eba22d8d2897439b06fa041100063858afdc0f83deef3c2f6a0a" gracePeriod=45
Sep 05 03:14:55.525554 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:14:55.525513    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" podUID="ac938c48-2041-4379-abe1-7e56c000d462" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:15:05.524535 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:05.524494    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" podUID="ac938c48-2041-4379-abe1-7e56c000d462" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:15:10.709349 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.709308    2568 generic.go:356] "Generic (PLEG): container finished" podID="ac938c48-2041-4379-abe1-7e56c000d462" containerID="23bb6acd77f0eba22d8d2897439b06fa041100063858afdc0f83deef3c2f6a0a" exitCode=0
Sep 05 03:15:10.709715 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.709394    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" event={"ID":"ac938c48-2041-4379-abe1-7e56c000d462","Type":"ContainerDied","Data":"23bb6acd77f0eba22d8d2897439b06fa041100063858afdc0f83deef3c2f6a0a"}
Sep 05 03:15:10.791416 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.791389    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:15:10.904418 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.904387    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k4hk\" (UniqueName: \"kubernetes.io/projected/ac938c48-2041-4379-abe1-7e56c000d462-kube-api-access-9k4hk\") pod \"ac938c48-2041-4379-abe1-7e56c000d462\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") "
Sep 05 03:15:10.904594 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.904459    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac938c48-2041-4379-abe1-7e56c000d462-tls-certs\") pod \"ac938c48-2041-4379-abe1-7e56c000d462\" (UID: \"ac938c48-2041-4379-abe1-7e56c000d462\") "
Sep 05 03:15:10.906736 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.906701    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac938c48-2041-4379-abe1-7e56c000d462-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac938c48-2041-4379-abe1-7e56c000d462" (UID: "ac938c48-2041-4379-abe1-7e56c000d462"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:15:10.906855 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:10.906747    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac938c48-2041-4379-abe1-7e56c000d462-kube-api-access-9k4hk" (OuterVolumeSpecName: "kube-api-access-9k4hk") pod "ac938c48-2041-4379-abe1-7e56c000d462" (UID: "ac938c48-2041-4379-abe1-7e56c000d462"). InnerVolumeSpecName "kube-api-access-9k4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:15:11.005394 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.005352    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4hk\" (UniqueName: \"kubernetes.io/projected/ac938c48-2041-4379-abe1-7e56c000d462-kube-api-access-9k4hk\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:15:11.005394 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.005386    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac938c48-2041-4379-abe1-7e56c000d462-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:15:11.714151 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.714118    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs" event={"ID":"ac938c48-2041-4379-abe1-7e56c000d462","Type":"ContainerDied","Data":"37140585087a5680540c7625c22ab2caa958c93b10f0d3e50d133938d9d11f2e"}
Sep 05 03:15:11.714668 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.714181    2568 scope.go:117] "RemoveContainer" containerID="23bb6acd77f0eba22d8d2897439b06fa041100063858afdc0f83deef3c2f6a0a"
Sep 05 03:15:11.714668 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.714124    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"
Sep 05 03:15:11.735573 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.735540    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"]
Sep 05 03:15:11.737110 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:11.737067    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddschrs"]
Sep 05 03:15:13.030372 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:15:13.030291    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac938c48-2041-4379-abe1-7e56c000d462" path="/var/lib/kubelet/pods/ac938c48-2041-4379-abe1-7e56c000d462/volumes"
Sep 05 03:17:41.126203 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.126162    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"]
Sep 05 03:17:41.128432 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.126462    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac938c48-2041-4379-abe1-7e56c000d462" containerName="main"
Sep 05 03:17:41.128432 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.126475    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac938c48-2041-4379-abe1-7e56c000d462" containerName="main"
Sep 05 03:17:41.128432 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.126535    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac938c48-2041-4379-abe1-7e56c000d462" containerName="main"
Sep 05 03:17:41.129236 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.129219    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.131802 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.131781    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 05 03:17:41.133243 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.133222    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 05 03:17:41.133394 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.133377    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-c4j24\""
Sep 05 03:17:41.133469 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.133415    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 05 03:17:41.133469 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.133447    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-rpdns\""
Sep 05 03:17:41.140084 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.140054    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"]
Sep 05 03:17:41.197858 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.197814    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60e4f72-6c4a-4bce-b91d-c76158a971b6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.198024 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.197917    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfkk\" (UniqueName: \"kubernetes.io/projected/a60e4f72-6c4a-4bce-b91d-c76158a971b6-kube-api-access-wpfkk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.298984 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.298947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfkk\" (UniqueName: \"kubernetes.io/projected/a60e4f72-6c4a-4bce-b91d-c76158a971b6-kube-api-access-wpfkk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.299170 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.299008    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60e4f72-6c4a-4bce-b91d-c76158a971b6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.301612 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.301591    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60e4f72-6c4a-4bce-b91d-c76158a971b6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.306591 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.306565    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfkk\" (UniqueName: \"kubernetes.io/projected/a60e4f72-6c4a-4bce-b91d-c76158a971b6-kube-api-access-wpfkk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.440014 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.439920    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:41.712654 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:41.712617    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"]
Sep 05 03:17:41.715171 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:17:41.715136    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60e4f72_6c4a_4bce_b91d_c76158a971b6.slice/crio-686b1b49359e3734f3df003f7beef1432395f480c8ac4901dac8c7860073ed55 WatchSource:0}: Error finding container 686b1b49359e3734f3df003f7beef1432395f480c8ac4901dac8c7860073ed55: Status 404 returned error can't find the container with id 686b1b49359e3734f3df003f7beef1432395f480c8ac4901dac8c7860073ed55
Sep 05 03:17:42.319699 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:42.319654    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" event={"ID":"a60e4f72-6c4a-4bce-b91d-c76158a971b6","Type":"ContainerStarted","Data":"ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce"}
Sep 05 03:17:42.320064 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:42.319716    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" event={"ID":"a60e4f72-6c4a-4bce-b91d-c76158a971b6","Type":"ContainerStarted","Data":"686b1b49359e3734f3df003f7beef1432395f480c8ac4901dac8c7860073ed55"}
Sep 05 03:17:42.320064 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:42.319753    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:17:42.337398 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:17:42.337345    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podStartSLOduration=1.337329715 podStartE2EDuration="1.337329715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:17:42.335372488 +0000 UTC m=+3561.762488715" watchObservedRunningTime="2026-09-05 03:17:42.337329715 +0000 UTC m=+3561.764445940"
Sep 05 03:18:13.326089 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:18:13.325991    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:19:53.323408 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:19:53.323328    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"]
Sep 05 03:19:53.326427 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:19:53.326378    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:20:03.326354 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:03.326316    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:20:07.271341 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.271309    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"]
Sep 05 03:20:07.274503 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.274485    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.277120 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.277098    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 05 03:20:07.277463 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.277446    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 05 03:20:07.277559 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.277458    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:20:07.277618 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.277605    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-dtt9d\""
Sep 05 03:20:07.277741 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.277722    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 05 03:20:07.278287 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.278236    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 05 03:20:07.287500 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.287480    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"]
Sep 05 03:20:07.323885 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.323855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.323994 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.323906    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.323994 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.323925    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.324141 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.324003    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.324141 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.324037    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96bd2478-c154-46bc-ac76-8a60220c3d9a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.324141 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.324066    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7mr\" (UniqueName: \"kubernetes.io/projected/96bd2478-c154-46bc-ac76-8a60220c3d9a-kube-api-access-st7mr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.324260 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.324138    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bd2478-c154-46bc-ac76-8a60220c3d9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.324260 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.324172    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425281 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425293    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96bd2478-c154-46bc-ac76-8a60220c3d9a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425330    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7mr\" (UniqueName: \"kubernetes.io/projected/96bd2478-c154-46bc-ac76-8a60220c3d9a-kube-api-access-st7mr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bd2478-c154-46bc-ac76-8a60220c3d9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425383    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425409    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425719 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425473    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425719 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425503    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425719 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425704    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425882 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425850    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.425927 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425881    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.426204 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.425990    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.426625 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.426603    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96bd2478-c154-46bc-ac76-8a60220c3d9a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.427847 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.427819    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.428123 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.428104    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bd2478-c154-46bc-ac76-8a60220c3d9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.433005 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.432987    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7mr\" (UniqueName: \"kubernetes.io/projected/96bd2478-c154-46bc-ac76-8a60220c3d9a-kube-api-access-st7mr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.584893 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.584807    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:07.861339 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.861314    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"]
Sep 05 03:20:07.863372 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:20:07.863345    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bd2478_c154_46bc_ac76_8a60220c3d9a.slice/crio-6828d8a840c95289ff3648841ed92c1c791fc219f453bbdbdc02df715988c4a1 WatchSource:0}: Error finding container 6828d8a840c95289ff3648841ed92c1c791fc219f453bbdbdc02df715988c4a1: Status 404 returned error can't find the container with id 6828d8a840c95289ff3648841ed92c1c791fc219f453bbdbdc02df715988c4a1
Sep 05 03:20:07.904017 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:07.903993    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" event={"ID":"96bd2478-c154-46bc-ac76-8a60220c3d9a","Type":"ContainerStarted","Data":"6828d8a840c95289ff3648841ed92c1c791fc219f453bbdbdc02df715988c4a1"}
Sep 05 03:20:08.343939 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:08.343857    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main" containerID="cri-o://ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce" gracePeriod=45
Sep 05 03:20:08.909733 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:08.909696    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" event={"ID":"96bd2478-c154-46bc-ac76-8a60220c3d9a","Type":"ContainerStarted","Data":"568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565"}
Sep 05 03:20:10.919303 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:10.919271    2568 generic.go:356] "Generic (PLEG): container finished" podID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerID="568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565" exitCode=0
Sep 05 03:20:10.919693 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:10.919343    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" event={"ID":"96bd2478-c154-46bc-ac76-8a60220c3d9a","Type":"ContainerDied","Data":"568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565"}
Sep 05 03:20:11.926625 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:11.926583    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" event={"ID":"96bd2478-c154-46bc-ac76-8a60220c3d9a","Type":"ContainerStarted","Data":"7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5"}
Sep 05 03:20:11.950780 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:11.950726    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podStartSLOduration=2.804756383 podStartE2EDuration="4.950712204s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.145955821s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:20:11.949187092 +0000 UTC m=+3711.376303308" watchObservedRunningTime="2026-09-05 03:20:11.950712204 +0000 UTC m=+3711.377828426"
Sep 05 03:20:13.326232 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:13.326194    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:20:13.326616 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:13.326302    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:20:17.585822 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:17.585789    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:17.586231 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:17.585851    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:20:17.587342 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:17.587310    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:20:23.325739 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:23.325700    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:20:27.585530 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:27.585484    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:20:33.325847 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:33.325806    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:20:37.585764 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:37.585724    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:20:38.585894 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.585871    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:20:38.691905 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.691876    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60e4f72-6c4a-4bce-b91d-c76158a971b6-tls-certs\") pod \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") "
Sep 05 03:20:38.692116 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.691933    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpfkk\" (UniqueName: \"kubernetes.io/projected/a60e4f72-6c4a-4bce-b91d-c76158a971b6-kube-api-access-wpfkk\") pod \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\" (UID: \"a60e4f72-6c4a-4bce-b91d-c76158a971b6\") "
Sep 05 03:20:38.694222 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.694193    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60e4f72-6c4a-4bce-b91d-c76158a971b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a60e4f72-6c4a-4bce-b91d-c76158a971b6" (UID: "a60e4f72-6c4a-4bce-b91d-c76158a971b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:20:38.694332 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.694313    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60e4f72-6c4a-4bce-b91d-c76158a971b6-kube-api-access-wpfkk" (OuterVolumeSpecName: "kube-api-access-wpfkk") pod "a60e4f72-6c4a-4bce-b91d-c76158a971b6" (UID: "a60e4f72-6c4a-4bce-b91d-c76158a971b6"). InnerVolumeSpecName "kube-api-access-wpfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:20:38.793145 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.793107    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60e4f72-6c4a-4bce-b91d-c76158a971b6-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:20:38.793145 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:38.793145    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfkk\" (UniqueName: \"kubernetes.io/projected/a60e4f72-6c4a-4bce-b91d-c76158a971b6-kube-api-access-wpfkk\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:20:39.043815 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.043781    2568 generic.go:356] "Generic (PLEG): container finished" podID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerID="ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce" exitCode=0
Sep 05 03:20:39.043988 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.043826    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"
Sep 05 03:20:39.043988 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.043863    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" event={"ID":"a60e4f72-6c4a-4bce-b91d-c76158a971b6","Type":"ContainerDied","Data":"ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce"}
Sep 05 03:20:39.043988 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.043911    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7" event={"ID":"a60e4f72-6c4a-4bce-b91d-c76158a971b6","Type":"ContainerDied","Data":"686b1b49359e3734f3df003f7beef1432395f480c8ac4901dac8c7860073ed55"}
Sep 05 03:20:39.043988 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.043933    2568 scope.go:117] "RemoveContainer" containerID="ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce"
Sep 05 03:20:39.052660 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.052643    2568 scope.go:117] "RemoveContainer" containerID="ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce"
Sep 05 03:20:39.052949 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:20:39.052919    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce\": container with ID starting with ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce not found: ID does not exist" containerID="ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce"
Sep 05 03:20:39.053021 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.052957    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce"} err="failed to get container status \"ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce\": rpc error: code = NotFound desc = could not find container \"ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce\": container with ID starting with ad66b169048847b1c02b71eda49f0dcdf916c14b61aaa24df20195ff6282c5ce not found: ID does not exist"
Sep 05 03:20:39.062042 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.062017    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"]
Sep 05 03:20:39.067884 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:39.067854    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pj2j7"]
Sep 05 03:20:41.031122 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:41.031059    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" path="/var/lib/kubelet/pods/a60e4f72-6c4a-4bce-b91d-c76158a971b6/volumes"
Sep 05 03:20:47.585898 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:47.585854    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:20:57.585524 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:20:57.585483    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:21:07.586049 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:21:07.585996    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:21:17.586249 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:21:17.586202    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:21:27.585796 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:21:27.585750    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:21:37.586273 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:21:37.586220    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:21:47.586170 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:21:47.586120    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:21:57.585524 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:21:57.585477    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 05 03:22:07.595036 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:07.595001    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:22:07.602996 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:07.602966    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:22:30.790709 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:30.790670    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"]
Sep 05 03:22:45.814552 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:45.814467    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main" containerID="cri-o://7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5" gracePeriod=45
Sep 05 03:22:46.552861 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.552837    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:22:46.591395 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.591319    2568 generic.go:356] "Generic (PLEG): container finished" podID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerID="7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5" exitCode=0
Sep 05 03:22:46.591395 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.591371    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"
Sep 05 03:22:46.591547 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.591367    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" event={"ID":"96bd2478-c154-46bc-ac76-8a60220c3d9a","Type":"ContainerDied","Data":"7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5"}
Sep 05 03:22:46.591547 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.591498    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr" event={"ID":"96bd2478-c154-46bc-ac76-8a60220c3d9a","Type":"ContainerDied","Data":"6828d8a840c95289ff3648841ed92c1c791fc219f453bbdbdc02df715988c4a1"}
Sep 05 03:22:46.591547 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.591523    2568 scope.go:117] "RemoveContainer" containerID="7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5"
Sep 05 03:22:46.599772 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.599754    2568 scope.go:117] "RemoveContainer" containerID="568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565"
Sep 05 03:22:46.606903 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.606886    2568 scope.go:117] "RemoveContainer" containerID="7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5"
Sep 05 03:22:46.607196 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:22:46.607170    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5\": container with ID starting with 7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5 not found: ID does not exist" containerID="7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5"
Sep 05 03:22:46.607272 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.607198    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5"} err="failed to get container status \"7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5\": rpc error: code = NotFound desc = could not find container \"7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5\": container with ID starting with 7dce3a9fa03c873152a64479fb37c7b73584e0087a0ba38c8882b2aa84c971c5 not found: ID does not exist"
Sep 05 03:22:46.607272 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.607214    2568 scope.go:117] "RemoveContainer" containerID="568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565"
Sep 05 03:22:46.607456 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:22:46.607442    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565\": container with ID starting with 568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565 not found: ID does not exist" containerID="568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565"
Sep 05 03:22:46.607493 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.607460    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565"} err="failed to get container status \"568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565\": rpc error: code = NotFound desc = could not find container \"568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565\": container with ID starting with 568431c32f84630c42f479f774f961ee7b102abd957b13679e2f1d7b05e78565 not found: ID does not exist"
Sep 05 03:22:46.622238 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622220    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-home\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622320 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622267    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-kserve-provision-location\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622320 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622304    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bd2478-c154-46bc-ac76-8a60220c3d9a-tls-certs\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622406 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622331    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-model-cache\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622406 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622366    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7mr\" (UniqueName: \"kubernetes.io/projected/96bd2478-c154-46bc-ac76-8a60220c3d9a-kube-api-access-st7mr\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622406 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622397    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-dshm\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622557 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622421    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-tmp-dir\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622557 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622444    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96bd2478-c154-46bc-ac76-8a60220c3d9a-cabundle-cert\") pod \"96bd2478-c154-46bc-ac76-8a60220c3d9a\" (UID: \"96bd2478-c154-46bc-ac76-8a60220c3d9a\") "
Sep 05 03:22:46.622661 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622622    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:46.622718 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622687    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.623037 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622978    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bd2478-c154-46bc-ac76-8a60220c3d9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:22:46.623037 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.622979    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-home" (OuterVolumeSpecName: "home") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:46.624767 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.624742    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bd2478-c154-46bc-ac76-8a60220c3d9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:22:46.624897 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.624869    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bd2478-c154-46bc-ac76-8a60220c3d9a-kube-api-access-st7mr" (OuterVolumeSpecName: "kube-api-access-st7mr") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "kube-api-access-st7mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:22:46.625124 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.625105    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-dshm" (OuterVolumeSpecName: "dshm") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:46.634745 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.634722    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:46.644064 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.644043    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96bd2478-c154-46bc-ac76-8a60220c3d9a" (UID: "96bd2478-c154-46bc-ac76-8a60220c3d9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:46.723590 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723561    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.723590 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723585    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.723749 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723595    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bd2478-c154-46bc-ac76-8a60220c3d9a-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.723749 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723606    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7mr\" (UniqueName: \"kubernetes.io/projected/96bd2478-c154-46bc-ac76-8a60220c3d9a-kube-api-access-st7mr\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.723749 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723613    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.723749 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723621    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bd2478-c154-46bc-ac76-8a60220c3d9a-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.723749 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.723628    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96bd2478-c154-46bc-ac76-8a60220c3d9a-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:22:46.919775 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.918655    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"]
Sep 05 03:22:46.922222 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:46.922196    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-66jllfr"]
Sep 05 03:22:47.030919 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:22:47.030886    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" path="/var/lib/kubelet/pods/96bd2478-c154-46bc-ac76-8a60220c3d9a/volumes"
Sep 05 03:23:36.166870 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.166820    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"]
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167114    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167126    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167139    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167145    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167154    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="storage-initializer"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167160    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="storage-initializer"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167212    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="96bd2478-c154-46bc-ac76-8a60220c3d9a" containerName="main"
Sep 05 03:23:36.167280 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.167220    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60e4f72-6c4a-4bce-b91d-c76158a971b6" containerName="main"
Sep 05 03:23:36.171227 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.171209    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.174510 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.174484    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 05 03:23:36.174510 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.174500    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 05 03:23:36.174667 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.174484    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 05 03:23:36.174667 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.174500    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 05 03:23:36.174667 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.174586    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-b2587\""
Sep 05 03:23:36.174667 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.174501    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:23:36.176414 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176517 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176484    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176597 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzw9z\" (UniqueName: \"kubernetes.io/projected/95da6a42-9995-488f-aea6-090b9dde28db-kube-api-access-kzw9z\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176597 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176553    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176597 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176582    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176759 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176608    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95da6a42-9995-488f-aea6-090b9dde28db-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176759 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.176759 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.176709    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95da6a42-9995-488f-aea6-090b9dde28db-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.185051 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.185031    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"]
Sep 05 03:23:36.277326 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277298    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95da6a42-9995-488f-aea6-090b9dde28db-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277326 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277335    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277559 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277366    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277559 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzw9z\" (UniqueName: \"kubernetes.io/projected/95da6a42-9995-488f-aea6-090b9dde28db-kube-api-access-kzw9z\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277559 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277498    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277559 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277526    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277559 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277548    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95da6a42-9995-488f-aea6-090b9dde28db-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277832 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277598    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.277832 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277737    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.278011 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.277977    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.278094 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.278040    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.278158 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.278043    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.278414 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.278390    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95da6a42-9995-488f-aea6-090b9dde28db-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.280033 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.280012    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95da6a42-9995-488f-aea6-090b9dde28db-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.280128 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.280041    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.285715 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.285693    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzw9z\" (UniqueName: \"kubernetes.io/projected/95da6a42-9995-488f-aea6-090b9dde28db-kube-api-access-kzw9z\") pod \"custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.481341 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.481226    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:36.762586 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.762506    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"]
Sep 05 03:23:36.766651 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:23:36.766616    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95da6a42_9995_488f_aea6_090b9dde28db.slice/crio-4289e4f8c92f1e61a7cbbdea25ade04bc17dad65e569249abf74289c911a5e64 WatchSource:0}: Error finding container 4289e4f8c92f1e61a7cbbdea25ade04bc17dad65e569249abf74289c911a5e64: Status 404 returned error can't find the container with id 4289e4f8c92f1e61a7cbbdea25ade04bc17dad65e569249abf74289c911a5e64
Sep 05 03:23:36.799604 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:36.799573    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" event={"ID":"95da6a42-9995-488f-aea6-090b9dde28db","Type":"ContainerStarted","Data":"4289e4f8c92f1e61a7cbbdea25ade04bc17dad65e569249abf74289c911a5e64"}
Sep 05 03:23:37.804574 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:37.804535    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" event={"ID":"95da6a42-9995-488f-aea6-090b9dde28db","Type":"ContainerStarted","Data":"ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c"}
Sep 05 03:23:39.815139 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:39.815097    2568 generic.go:356] "Generic (PLEG): container finished" podID="95da6a42-9995-488f-aea6-090b9dde28db" containerID="ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c" exitCode=0
Sep 05 03:23:39.815510 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:39.815161    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" event={"ID":"95da6a42-9995-488f-aea6-090b9dde28db","Type":"ContainerDied","Data":"ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c"}
Sep 05 03:23:40.821919 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:40.821878    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" event={"ID":"95da6a42-9995-488f-aea6-090b9dde28db","Type":"ContainerStarted","Data":"ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8"}
Sep 05 03:23:40.847236 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:40.847165    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podStartSLOduration=2.665528673 podStartE2EDuration="4.847150306s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.181621633s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:23:40.845434908 +0000 UTC m=+3920.272551137" watchObservedRunningTime="2026-09-05 03:23:40.847150306 +0000 UTC m=+3920.274266531"
Sep 05 03:23:46.482218 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:46.482177    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:46.482637 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:46.482241    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:23:46.483705 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:46.483677    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:23:56.482263 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:23:56.482222    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:24:06.481910 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:24:06.481869    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:24:16.482161 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:24:16.482111    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:24:26.482607 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:24:26.482556    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:24:36.482460 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:24:36.482414    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:24:46.482504 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:24:46.482451    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:24:56.482631 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:24:56.482582    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:25:06.482319 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:25:06.482266    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:25:16.482061 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:25:16.482013    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:25:26.481892 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:25:26.481844    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 05 03:25:36.492142 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:25:36.492112    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:25:36.499875 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:25:36.499847    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:26:34.729305 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:34.729270    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"]
Sep 05 03:26:49.752904 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:49.752842    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main" containerID="cri-o://ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8" gracePeriod=45
Sep 05 03:26:50.420618 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.420595    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:26:50.513404 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513337    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95da6a42-9995-488f-aea6-090b9dde28db-cabundle-cert\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513404 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513373    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-tmp-dir\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513404 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513400    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95da6a42-9995-488f-aea6-090b9dde28db-tls-certs\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513658 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513636    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-model-cache\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513718 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513686    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-home\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513781 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513717    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95da6a42-9995-488f-aea6-090b9dde28db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:26:50.513781 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513770    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-kserve-provision-location\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513886 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513821    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-dshm\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513886 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513861    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzw9z\" (UniqueName: \"kubernetes.io/projected/95da6a42-9995-488f-aea6-090b9dde28db-kube-api-access-kzw9z\") pod \"95da6a42-9995-488f-aea6-090b9dde28db\" (UID: \"95da6a42-9995-488f-aea6-090b9dde28db\") "
Sep 05 03:26:50.513986 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.513910    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-model-cache" (OuterVolumeSpecName: "model-cache") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.514160 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.514116    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.514160 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.514138    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95da6a42-9995-488f-aea6-090b9dde28db-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.514310 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.514268    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-home" (OuterVolumeSpecName: "home") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.516326 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.516295    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95da6a42-9995-488f-aea6-090b9dde28db-kube-api-access-kzw9z" (OuterVolumeSpecName: "kube-api-access-kzw9z") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "kube-api-access-kzw9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:26:50.516473 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.516449    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95da6a42-9995-488f-aea6-090b9dde28db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:26:50.516598 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.516503    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-dshm" (OuterVolumeSpecName: "dshm") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.525028 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.525007    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.541621 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.541597    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95da6a42-9995-488f-aea6-090b9dde28db" (UID: "95da6a42-9995-488f-aea6-090b9dde28db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.615086 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.615044    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.615223 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.615089    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.615223 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.615104    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzw9z\" (UniqueName: \"kubernetes.io/projected/95da6a42-9995-488f-aea6-090b9dde28db-kube-api-access-kzw9z\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.615223 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.615116    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.615223 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.615127    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95da6a42-9995-488f-aea6-090b9dde28db-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.615223 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.615137    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95da6a42-9995-488f-aea6-090b9dde28db-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.619912 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.619886    2568 generic.go:356] "Generic (PLEG): container finished" podID="95da6a42-9995-488f-aea6-090b9dde28db" containerID="ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8" exitCode=0
Sep 05 03:26:50.620033 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.619923    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" event={"ID":"95da6a42-9995-488f-aea6-090b9dde28db","Type":"ContainerDied","Data":"ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8"}
Sep 05 03:26:50.620033 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.619944    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"
Sep 05 03:26:50.620033 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.619992    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2" event={"ID":"95da6a42-9995-488f-aea6-090b9dde28db","Type":"ContainerDied","Data":"4289e4f8c92f1e61a7cbbdea25ade04bc17dad65e569249abf74289c911a5e64"}
Sep 05 03:26:50.620033 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.620015    2568 scope.go:117] "RemoveContainer" containerID="ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8"
Sep 05 03:26:50.628923 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.628908    2568 scope.go:117] "RemoveContainer" containerID="ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c"
Sep 05 03:26:50.636547 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.636531    2568 scope.go:117] "RemoveContainer" containerID="ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8"
Sep 05 03:26:50.636813 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:26:50.636792    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8\": container with ID starting with ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8 not found: ID does not exist" containerID="ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8"
Sep 05 03:26:50.636888 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.636826    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8"} err="failed to get container status \"ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8\": rpc error: code = NotFound desc = could not find container \"ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8\": container with ID starting with ff690b3f26bf452991b3f1e9ed14dfdd8e12ef34db5cb15ba92aae309d6d26a8 not found: ID does not exist"
Sep 05 03:26:50.636888 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.636850    2568 scope.go:117] "RemoveContainer" containerID="ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c"
Sep 05 03:26:50.637152 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:26:50.637134    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c\": container with ID starting with ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c not found: ID does not exist" containerID="ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c"
Sep 05 03:26:50.637196 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.637157    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c"} err="failed to get container status \"ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c\": rpc error: code = NotFound desc = could not find container \"ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c\": container with ID starting with ca409a2fc9fd83f89183eba128127e0b3eb8bd77c1b13df60620f96eddcd462c not found: ID does not exist"
Sep 05 03:26:50.641633 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.641612    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"]
Sep 05 03:26:50.647888 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:50.647870    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-749cc96dcc-tzqp2"]
Sep 05 03:26:51.030411 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:26:51.030378    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95da6a42-9995-488f-aea6-090b9dde28db" path="/var/lib/kubelet/pods/95da6a42-9995-488f-aea6-090b9dde28db/volumes"
Sep 05 03:27:51.467797 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.467760    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"]
Sep 05 03:27:51.470325 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.468058    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="storage-initializer"
Sep 05 03:27:51.470325 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.468087    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="storage-initializer"
Sep 05 03:27:51.470325 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.468099    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main"
Sep 05 03:27:51.470325 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.468108    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main"
Sep 05 03:27:51.470325 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.468178    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="95da6a42-9995-488f-aea6-090b9dde28db" containerName="main"
Sep 05 03:27:51.471355 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.471332    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.474929 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.474904    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 05 03:27:51.475099 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.475053    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 05 03:27:51.475223 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.475123    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:27:51.475316 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.475294    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 05 03:27:51.475519 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.475505    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-27csp\""
Sep 05 03:27:51.475894 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.475872    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 05 03:27:51.485494 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.485473    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"]
Sep 05 03:27:51.551476 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551443    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551642 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551495    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551642 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551560    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncg9\" (UniqueName: \"kubernetes.io/projected/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kube-api-access-6ncg9\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551642 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551755 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551647    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-home\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551755 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551664    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551755 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551684    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.551755 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.551719    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652515 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652483    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-home\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652515 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652515    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652720 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652535    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652762 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652802 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652788    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652847 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652826    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652892 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652849    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncg9\" (UniqueName: \"kubernetes.io/projected/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kube-api-access-6ncg9\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.652892 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.652866    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.653281 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.653201    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.653281 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.653247    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.653281 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.653273    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.653486 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.653276    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-home\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.653565 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.653545    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.654978 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.654957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.655418 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.655399    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.660938 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.660913    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncg9\" (UniqueName: \"kubernetes.io/projected/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kube-api-access-6ncg9\") pod \"router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:51.782176 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:51.782092    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:27:52.060786 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:52.060754    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"]
Sep 05 03:27:52.062733 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:27:52.062702    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf70fe12_5a29_435c_a6c1_a2d710cc98ef.slice/crio-721b8209410f858068d6eaf9a384ec9f6415f182f69b1059f75c1cde2a6f434c WatchSource:0}: Error finding container 721b8209410f858068d6eaf9a384ec9f6415f182f69b1059f75c1cde2a6f434c: Status 404 returned error can't find the container with id 721b8209410f858068d6eaf9a384ec9f6415f182f69b1059f75c1cde2a6f434c
Sep 05 03:27:52.877734 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:52.877695    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" event={"ID":"cf70fe12-5a29-435c-a6c1-a2d710cc98ef","Type":"ContainerStarted","Data":"47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8"}
Sep 05 03:27:52.878307 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:52.877745    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" event={"ID":"cf70fe12-5a29-435c-a6c1-a2d710cc98ef","Type":"ContainerStarted","Data":"721b8209410f858068d6eaf9a384ec9f6415f182f69b1059f75c1cde2a6f434c"}
Sep 05 03:27:54.887492 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:54.887455    2568 generic.go:356] "Generic (PLEG): container finished" podID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerID="47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8" exitCode=0
Sep 05 03:27:54.887861 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:54.887532    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" event={"ID":"cf70fe12-5a29-435c-a6c1-a2d710cc98ef","Type":"ContainerDied","Data":"47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8"}
Sep 05 03:27:55.894011 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:55.893978    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" event={"ID":"cf70fe12-5a29-435c-a6c1-a2d710cc98ef","Type":"ContainerStarted","Data":"3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf"}
Sep 05 03:27:55.915832 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:27:55.915766    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podStartSLOduration=2.781358821 podStartE2EDuration="4.915745661s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.13438684s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:27:55.914404051 +0000 UTC m=+4175.341520275" watchObservedRunningTime="2026-09-05 03:27:55.915745661 +0000 UTC m=+4175.342861886"
Sep 05 03:28:01.782465 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:01.782424    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:28:01.782848 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:01.782494    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:28:01.783728 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:01.783699    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:28:11.782599 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:11.782537    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:28:21.783471 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:21.783422    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:28:31.782913 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:31.782871    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:28:41.783001 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:41.782954    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:28:51.783227 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:28:51.783134    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:29:01.782597 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:01.782555    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:29:11.783021 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:11.782975    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:29:21.783214 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:21.783170    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:29:31.782696 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:31.782655    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:29:41.782939 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:41.782888    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 05 03:29:51.793201 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:51.793170    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:29:51.801277 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:29:51.801253    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:30:04.650612 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:04.650575    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"]
Sep 05 03:30:19.674350 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:19.674287    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main" containerID="cri-o://3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf" gracePeriod=45
Sep 05 03:30:20.328393 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.328368    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:30:20.448880 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448797    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-cabundle-cert\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.448880 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448845    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tls-certs\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.448880 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448864    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tmp-dir\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.448880 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448882    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-dshm\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.449310 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448900    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-model-cache\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.449310 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448939    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ncg9\" (UniqueName: \"kubernetes.io/projected/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kube-api-access-6ncg9\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.449310 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.448966    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kserve-provision-location\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.449310 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.449011    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-home\") pod \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\" (UID: \"cf70fe12-5a29-435c-a6c1-a2d710cc98ef\") "
Sep 05 03:30:20.449523 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.449302    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.449523 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.449478    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:30:20.449723 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.449705    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-home" (OuterVolumeSpecName: "home") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.451931 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.451905    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:30:20.451931 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.451919    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-dshm" (OuterVolumeSpecName: "dshm") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.452119 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.451932    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kube-api-access-6ncg9" (OuterVolumeSpecName: "kube-api-access-6ncg9") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "kube-api-access-6ncg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:30:20.462217 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.462062    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.472272 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.472245    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf70fe12-5a29-435c-a6c1-a2d710cc98ef" (UID: "cf70fe12-5a29-435c-a6c1-a2d710cc98ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.507426 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.507396    2568 generic.go:356] "Generic (PLEG): container finished" podID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerID="3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf" exitCode=0
Sep 05 03:30:20.507551 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.507452    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"
Sep 05 03:30:20.507551 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.507469    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" event={"ID":"cf70fe12-5a29-435c-a6c1-a2d710cc98ef","Type":"ContainerDied","Data":"3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf"}
Sep 05 03:30:20.507551 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.507525    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k" event={"ID":"cf70fe12-5a29-435c-a6c1-a2d710cc98ef","Type":"ContainerDied","Data":"721b8209410f858068d6eaf9a384ec9f6415f182f69b1059f75c1cde2a6f434c"}
Sep 05 03:30:20.507551 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.507551    2568 scope.go:117] "RemoveContainer" containerID="3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf"
Sep 05 03:30:20.516260 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.516242    2568 scope.go:117] "RemoveContainer" containerID="47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8"
Sep 05 03:30:20.523754 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.523736    2568 scope.go:117] "RemoveContainer" containerID="3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf"
Sep 05 03:30:20.524030 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:30:20.524009    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf\": container with ID starting with 3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf not found: ID does not exist" containerID="3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf"
Sep 05 03:30:20.524097 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.524038    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf"} err="failed to get container status \"3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf\": rpc error: code = NotFound desc = could not find container \"3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf\": container with ID starting with 3087d5da416ed113703067163d5982038a7766d9bf67afbd8dcd3dd825883fcf not found: ID does not exist"
Sep 05 03:30:20.524097 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.524055    2568 scope.go:117] "RemoveContainer" containerID="47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8"
Sep 05 03:30:20.524341 ip-10-0-140-68 kubenswrapper[2568]: E0905 03:30:20.524322    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8\": container with ID starting with 47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8 not found: ID does not exist" containerID="47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8"
Sep 05 03:30:20.524401 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.524350    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8"} err="failed to get container status \"47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8\": rpc error: code = NotFound desc = could not find container \"47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8\": container with ID starting with 47701196ce307601ac1eb3b405e2037ad30f88aeb64be185deb17c9a812419a8 not found: ID does not exist"
Sep 05 03:30:20.528263 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.528227    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"]
Sep 05 03:30:20.532482 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.532463    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59dc694f-wzp6k"]
Sep 05 03:30:20.549775 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549752    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncg9\" (UniqueName: \"kubernetes.io/projected/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kube-api-access-6ncg9\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549775 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549774    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549784    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549794    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549802    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549809    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549816    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.549877 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:20.549823    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf70fe12-5a29-435c-a6c1-a2d710cc98ef-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:30:21.035544 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:30:21.035507    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" path="/var/lib/kubelet/pods/cf70fe12-5a29-435c-a6c1-a2d710cc98ef/volumes"
Sep 05 03:31:01.902600 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.902567    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"]
Sep 05 03:31:01.903004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.902863    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main"
Sep 05 03:31:01.903004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.902874    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main"
Sep 05 03:31:01.903004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.902912    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="storage-initializer"
Sep 05 03:31:01.903004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.902918    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="storage-initializer"
Sep 05 03:31:01.903004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.902977    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf70fe12-5a29-435c-a6c1-a2d710cc98ef" containerName="main"
Sep 05 03:31:01.905904 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.905884    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:01.909524 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.909501    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-j5gxv\""
Sep 05 03:31:01.909645 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.909509    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 05 03:31:01.909645 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.909590    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 05 03:31:01.909645 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.909614    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-4jbvk\""
Sep 05 03:31:01.909898 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.909876    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 05 03:31:01.915619 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.915597    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"]
Sep 05 03:31:01.941481 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.941454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a31c7fd-fd94-4890-be37-904b0e5beb96-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:01.941600 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:01.941493    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4n8\" (UniqueName: \"kubernetes.io/projected/1a31c7fd-fd94-4890-be37-904b0e5beb96-kube-api-access-vz4n8\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.042324 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.042285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a31c7fd-fd94-4890-be37-904b0e5beb96-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.042568 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.042347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4n8\" (UniqueName: \"kubernetes.io/projected/1a31c7fd-fd94-4890-be37-904b0e5beb96-kube-api-access-vz4n8\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.045562 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.045538    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a31c7fd-fd94-4890-be37-904b0e5beb96-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.050694 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.050670    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4n8\" (UniqueName: \"kubernetes.io/projected/1a31c7fd-fd94-4890-be37-904b0e5beb96-kube-api-access-vz4n8\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.216746 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.216660    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.489441 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.489417    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"]
Sep 05 03:31:02.491878 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:31:02.491844    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a31c7fd_fd94_4890_be37_904b0e5beb96.slice/crio-2b3ef9867a22b5ed61a93e30899cda1c7b12ea92c0238d1df2dc89379f1ac84d WatchSource:0}: Error finding container 2b3ef9867a22b5ed61a93e30899cda1c7b12ea92c0238d1df2dc89379f1ac84d: Status 404 returned error can't find the container with id 2b3ef9867a22b5ed61a93e30899cda1c7b12ea92c0238d1df2dc89379f1ac84d
Sep 05 03:31:02.678967 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.678932    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" event={"ID":"1a31c7fd-fd94-4890-be37-904b0e5beb96","Type":"ContainerStarted","Data":"a818a392a11d6bde01c73767f5515064a0866d76574fc227b7f8ad716990ae1b"}
Sep 05 03:31:02.679173 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.678988    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" event={"ID":"1a31c7fd-fd94-4890-be37-904b0e5beb96","Type":"ContainerStarted","Data":"2b3ef9867a22b5ed61a93e30899cda1c7b12ea92c0238d1df2dc89379f1ac84d"}
Sep 05 03:31:02.679173 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.679016    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:02.701565 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:02.701512    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" podStartSLOduration=1.701496881 podStartE2EDuration="1.701496881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:31:02.699843265 +0000 UTC m=+4362.126959490" watchObservedRunningTime="2026-09-05 03:31:02.701496881 +0000 UTC m=+4362.128613105"
Sep 05 03:31:33.684521 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:33.684493    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:31:41.313049 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.313008    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"]
Sep 05 03:31:41.316184 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.316168    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.319258 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.319227    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 05 03:31:41.319390 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.319289    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 05 03:31:41.319390 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.319317    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:31:41.319496 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.319457    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-8g5ll\""
Sep 05 03:31:41.320388 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.320371    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 05 03:31:41.320475 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.320395    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 05 03:31:41.329459 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.329434    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"]
Sep 05 03:31:41.450068 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450040    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-tmp-dir\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450247 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450102    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-dshm\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450247 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450142    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-home\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450247 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450180    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c75ca1-063e-4828-b17b-41f5c8e418b5-tls-certs\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450247 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5g6\" (UniqueName: \"kubernetes.io/projected/82c75ca1-063e-4828-b17b-41f5c8e418b5-kube-api-access-9l5g6\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450247 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450219    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-kserve-provision-location\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450290    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-model-cache\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.450420 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.450324    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82c75ca1-063e-4828-b17b-41f5c8e418b5-cabundle-cert\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551539 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551487    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c75ca1-063e-4828-b17b-41f5c8e418b5-tls-certs\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551539 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5g6\" (UniqueName: \"kubernetes.io/projected/82c75ca1-063e-4828-b17b-41f5c8e418b5-kube-api-access-9l5g6\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551803 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551577    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-kserve-provision-location\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551803 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-model-cache\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551803 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551640    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82c75ca1-063e-4828-b17b-41f5c8e418b5-cabundle-cert\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551803 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551683    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-tmp-dir\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551803 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551712    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-dshm\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.551803 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.551760    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-home\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.552213 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.552181    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-tmp-dir\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.552324 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.552211    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-model-cache\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.552324 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.552285    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-home\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.552425 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.552405    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-kserve-provision-location\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.552589 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.552573    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82c75ca1-063e-4828-b17b-41f5c8e418b5-cabundle-cert\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.554158 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.554138    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-dshm\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.554369 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.554349    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c75ca1-063e-4828-b17b-41f5c8e418b5-tls-certs\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.559738 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.559718    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5g6\" (UniqueName: \"kubernetes.io/projected/82c75ca1-063e-4828-b17b-41f5c8e418b5-kube-api-access-9l5g6\") pod \"prestop-hook-test-kserve-7787c46b47-nccdw\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.625911 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.625879    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:41.908301 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:41.908275    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"]
Sep 05 03:31:41.910515 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:31:41.910487    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c75ca1_063e_4828_b17b_41f5c8e418b5.slice/crio-2b4ffffa89e3a541d01706b5fc4a5715d80987958882d777efed0286ef0606e5 WatchSource:0}: Error finding container 2b4ffffa89e3a541d01706b5fc4a5715d80987958882d777efed0286ef0606e5: Status 404 returned error can't find the container with id 2b4ffffa89e3a541d01706b5fc4a5715d80987958882d777efed0286ef0606e5
Sep 05 03:31:42.847313 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:42.847276    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" event={"ID":"82c75ca1-063e-4828-b17b-41f5c8e418b5","Type":"ContainerStarted","Data":"48509c0e1680393c13aca4e5c7c9bd21c11050833783cb46c503f6833a48b8cc"}
Sep 05 03:31:42.847675 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:42.847363    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" event={"ID":"82c75ca1-063e-4828-b17b-41f5c8e418b5","Type":"ContainerStarted","Data":"2b4ffffa89e3a541d01706b5fc4a5715d80987958882d777efed0286ef0606e5"}
Sep 05 03:31:44.857151 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:44.857118    2568 generic.go:356] "Generic (PLEG): container finished" podID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerID="48509c0e1680393c13aca4e5c7c9bd21c11050833783cb46c503f6833a48b8cc" exitCode=0
Sep 05 03:31:44.857585 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:44.857170    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" event={"ID":"82c75ca1-063e-4828-b17b-41f5c8e418b5","Type":"ContainerDied","Data":"48509c0e1680393c13aca4e5c7c9bd21c11050833783cb46c503f6833a48b8cc"}
Sep 05 03:31:45.863925 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:45.863887    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" event={"ID":"82c75ca1-063e-4828-b17b-41f5c8e418b5","Type":"ContainerStarted","Data":"e069757003c801b4cec8c3f685a34c5170d28cd9264b087e6a43fc697293d0e5"}
Sep 05 03:31:45.888443 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:45.888386    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" podStartSLOduration=2.760293249 podStartE2EDuration="4.888368316s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.128075067s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:31:45.88575269 +0000 UTC m=+4405.312868928" watchObservedRunningTime="2026-09-05 03:31:45.888368316 +0000 UTC m=+4405.315484545"
Sep 05 03:31:46.626350 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:46.626312    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:51.626092 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:51.626042    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:31:51.627531 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:51.627503    2568 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 05 03:31:56.906352 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:56.906309    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"]
Sep 05 03:31:58.803819 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:31:58.803777    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"]
Sep 05 03:32:03.684532 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:03.684491    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:11.927458 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:11.927394    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main" containerID="cri-o://a818a392a11d6bde01c73767f5515064a0866d76574fc227b7f8ad716990ae1b" gracePeriod=45
Sep 05 03:32:13.684319 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:13.684279    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:13.829328 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:13.829274    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="main" containerID="cri-o://e069757003c801b4cec8c3f685a34c5170d28cd9264b087e6a43fc697293d0e5" gracePeriod=45
Sep 05 03:32:23.684285 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:23.684246    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:23.684678 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:23.684386    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:32:33.685696 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:33.685643    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:42.120774 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.120742    2568 generic.go:356] "Generic (PLEG): container finished" podID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerID="a818a392a11d6bde01c73767f5515064a0866d76574fc227b7f8ad716990ae1b" exitCode=0
Sep 05 03:32:42.121203 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.120819    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" event={"ID":"1a31c7fd-fd94-4890-be37-904b0e5beb96","Type":"ContainerDied","Data":"a818a392a11d6bde01c73767f5515064a0866d76574fc227b7f8ad716990ae1b"}
Sep 05 03:32:42.177277 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.177255    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:32:42.367275 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.367249    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a31c7fd-fd94-4890-be37-904b0e5beb96-tls-certs\") pod \"1a31c7fd-fd94-4890-be37-904b0e5beb96\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") "
Sep 05 03:32:42.367446 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.367294    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz4n8\" (UniqueName: \"kubernetes.io/projected/1a31c7fd-fd94-4890-be37-904b0e5beb96-kube-api-access-vz4n8\") pod \"1a31c7fd-fd94-4890-be37-904b0e5beb96\" (UID: \"1a31c7fd-fd94-4890-be37-904b0e5beb96\") "
Sep 05 03:32:42.369539 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.369511    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a31c7fd-fd94-4890-be37-904b0e5beb96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a31c7fd-fd94-4890-be37-904b0e5beb96" (UID: "1a31c7fd-fd94-4890-be37-904b0e5beb96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:32:42.369653 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.369582    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a31c7fd-fd94-4890-be37-904b0e5beb96-kube-api-access-vz4n8" (OuterVolumeSpecName: "kube-api-access-vz4n8") pod "1a31c7fd-fd94-4890-be37-904b0e5beb96" (UID: "1a31c7fd-fd94-4890-be37-904b0e5beb96"). InnerVolumeSpecName "kube-api-access-vz4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:32:42.468063 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.468033    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4n8\" (UniqueName: \"kubernetes.io/projected/1a31c7fd-fd94-4890-be37-904b0e5beb96-kube-api-access-vz4n8\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:42.468063 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:42.468058    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a31c7fd-fd94-4890-be37-904b0e5beb96-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:43.126336 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:43.126305    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv" event={"ID":"1a31c7fd-fd94-4890-be37-904b0e5beb96","Type":"ContainerDied","Data":"2b3ef9867a22b5ed61a93e30899cda1c7b12ea92c0238d1df2dc89379f1ac84d"}
Sep 05 03:32:43.126764 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:43.126364    2568 scope.go:117] "RemoveContainer" containerID="a818a392a11d6bde01c73767f5515064a0866d76574fc227b7f8ad716990ae1b"
Sep 05 03:32:43.126764 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:43.126310    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"
Sep 05 03:32:43.141230 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:43.141207    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"]
Sep 05 03:32:43.145153 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:43.145125    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-2k4vv"]
Sep 05 03:32:45.029946 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:45.029909    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" path="/var/lib/kubelet/pods/1a31c7fd-fd94-4890-be37-904b0e5beb96/volumes"
Sep 05 03:32:59.200812 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.200737    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7787c46b47-nccdw_82c75ca1-063e-4828-b17b-41f5c8e418b5/main/0.log"
Sep 05 03:32:59.201265 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.201141    2568 generic.go:356] "Generic (PLEG): container finished" podID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerID="e069757003c801b4cec8c3f685a34c5170d28cd9264b087e6a43fc697293d0e5" exitCode=137
Sep 05 03:32:59.201265 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.201209    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" event={"ID":"82c75ca1-063e-4828-b17b-41f5c8e418b5","Type":"ContainerDied","Data":"e069757003c801b4cec8c3f685a34c5170d28cd9264b087e6a43fc697293d0e5"}
Sep 05 03:32:59.463182 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.463154    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7787c46b47-nccdw_82c75ca1-063e-4828-b17b-41f5c8e418b5/main/0.log"
Sep 05 03:32:59.465716 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.465694    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:32:59.499840 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.499813    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-home\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500001 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.499854    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c75ca1-063e-4828-b17b-41f5c8e418b5-tls-certs\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500001 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.499884    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82c75ca1-063e-4828-b17b-41f5c8e418b5-cabundle-cert\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500001 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.499914    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-dshm\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500001 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.499935    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-model-cache\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500001 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.499969    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l5g6\" (UniqueName: \"kubernetes.io/projected/82c75ca1-063e-4828-b17b-41f5c8e418b5-kube-api-access-9l5g6\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500291 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.500002    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-tmp-dir\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500291 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.500042    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-kserve-provision-location\") pod \"82c75ca1-063e-4828-b17b-41f5c8e418b5\" (UID: \"82c75ca1-063e-4828-b17b-41f5c8e418b5\") "
Sep 05 03:32:59.500397 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.500350    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-home" (OuterVolumeSpecName: "home") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:32:59.500397 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.500375    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:32:59.500621 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.500595    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:32:59.500941 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.500898    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c75ca1-063e-4828-b17b-41f5c8e418b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:32:59.502717 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.502694    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c75ca1-063e-4828-b17b-41f5c8e418b5-kube-api-access-9l5g6" (OuterVolumeSpecName: "kube-api-access-9l5g6") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "kube-api-access-9l5g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:32:59.502808 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.502740    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c75ca1-063e-4828-b17b-41f5c8e418b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:32:59.502965 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.502947    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-dshm" (OuterVolumeSpecName: "dshm") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:32:59.521553 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.521531    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82c75ca1-063e-4828-b17b-41f5c8e418b5" (UID: "82c75ca1-063e-4828-b17b-41f5c8e418b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:32:59.601406 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601382    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-home\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601406 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601403    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c75ca1-063e-4828-b17b-41f5c8e418b5-tls-certs\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601546 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601413    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82c75ca1-063e-4828-b17b-41f5c8e418b5-cabundle-cert\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601546 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601422    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-dshm\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601546 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601430    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-model-cache\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601546 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601438    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l5g6\" (UniqueName: \"kubernetes.io/projected/82c75ca1-063e-4828-b17b-41f5c8e418b5-kube-api-access-9l5g6\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601546 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601446    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-tmp-dir\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:32:59.601546 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:32:59.601455    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82c75ca1-063e-4828-b17b-41f5c8e418b5-kserve-provision-location\") on node \"ip-10-0-140-68.ec2.internal\" DevicePath \"\""
Sep 05 03:33:00.206683 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.206653    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7787c46b47-nccdw_82c75ca1-063e-4828-b17b-41f5c8e418b5/main/0.log"
Sep 05 03:33:00.207123 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.207104    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"
Sep 05 03:33:00.207123 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.207103    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw" event={"ID":"82c75ca1-063e-4828-b17b-41f5c8e418b5","Type":"ContainerDied","Data":"2b4ffffa89e3a541d01706b5fc4a5715d80987958882d777efed0286ef0606e5"}
Sep 05 03:33:00.207210 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.207168    2568 scope.go:117] "RemoveContainer" containerID="e069757003c801b4cec8c3f685a34c5170d28cd9264b087e6a43fc697293d0e5"
Sep 05 03:33:00.216026 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.216005    2568 scope.go:117] "RemoveContainer" containerID="48509c0e1680393c13aca4e5c7c9bd21c11050833783cb46c503f6833a48b8cc"
Sep 05 03:33:00.228968 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.228919    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"]
Sep 05 03:33:00.232453 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:00.232425    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-nccdw"]
Sep 05 03:33:01.030628 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:01.030582    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" path="/var/lib/kubelet/pods/82c75ca1-063e-4828-b17b-41f5c8e418b5/volumes"
Sep 05 03:33:27.913384 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:27.913348    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tc4x8_91744050-b528-43d9-8bdb-8a2e67993ccb/discovery/0.log"
Sep 05 03:33:29.148261 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:29.148226    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tc4x8_91744050-b528-43d9-8bdb-8a2e67993ccb/discovery/0.log"
Sep 05 03:33:30.337691 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:30.337662    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-wcv4l_7504a7b0-eea3-4056-973e-eb684b34d21b/kuadrant-console-plugin/0.log"
Sep 05 03:33:39.370767 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:39.370697    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pvm72_c869803e-5bb5-4227-a6e8-07c97cf8193f/global-pull-secret-syncer/0.log"
Sep 05 03:33:39.479126 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:39.479096    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pkhz9_829859f0-9e5a-4624-bc4c-ab78a0d0d2f3/konnectivity-agent/0.log"
Sep 05 03:33:39.537159 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:39.537127    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-68.ec2.internal_63872a04a76bb538711c93178fe3d471/haproxy/0.log"
Sep 05 03:33:44.489865 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:44.489835    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-wcv4l_7504a7b0-eea3-4056-973e-eb684b34d21b/kuadrant-console-plugin/0.log"
Sep 05 03:33:46.649187 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:46.649158    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-7mqqs_9c8ab5ec-55dd-4ad4-8774-85d5a8893aae/cluster-monitoring-operator/0.log"
Sep 05 03:33:46.883360 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:46.883318    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpcbc_61e976bb-df8e-4d40-baa7-08bcee923fa7/node-exporter/0.log"
Sep 05 03:33:46.904349 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:46.904268    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpcbc_61e976bb-df8e-4d40-baa7-08bcee923fa7/kube-rbac-proxy/0.log"
Sep 05 03:33:46.923266 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:46.923245    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpcbc_61e976bb-df8e-4d40-baa7-08bcee923fa7/init-textfile/0.log"
Sep 05 03:33:50.017269 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017237    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"]
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017622    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="main"
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017640    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="main"
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017663    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main"
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017673    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main"
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017686    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="storage-initializer"
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017695    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="storage-initializer"
Sep 05 03:33:50.017766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017767    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a31c7fd-fd94-4890-be37-904b0e5beb96" containerName="main"
Sep 05 03:33:50.018150 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.017781    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c75ca1-063e-4828-b17b-41f5c8e418b5" containerName="main"
Sep 05 03:33:50.021005 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.020982    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.023306 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.023283    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g75mm\"/\"kube-root-ca.crt\""
Sep 05 03:33:50.024328 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.024304    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g75mm\"/\"openshift-service-ca.crt\""
Sep 05 03:33:50.024422 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.024347    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g75mm\"/\"default-dockercfg-js4pb\""
Sep 05 03:33:50.030777 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.030754    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"]
Sep 05 03:33:50.187680 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.187646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-proc\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.187859 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.187700    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-sys\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.187859 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.187752    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-lib-modules\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.187859 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.187784    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-podres\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.188036 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.187872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn529\" (UniqueName: \"kubernetes.io/projected/9d404619-cbb2-47cb-878b-ce9ee89b8d66-kube-api-access-pn529\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289024 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.288938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-proc\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289024 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.288982    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-sys\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289024 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289007    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-lib-modules\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289334 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-podres\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289334 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289061    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn529\" (UniqueName: \"kubernetes.io/projected/9d404619-cbb2-47cb-878b-ce9ee89b8d66-kube-api-access-pn529\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289334 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289094    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-sys\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289334 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289091    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-proc\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289334 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-lib-modules\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.289334 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.289158    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d404619-cbb2-47cb-878b-ce9ee89b8d66-podres\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.297578 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.297547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn529\" (UniqueName: \"kubernetes.io/projected/9d404619-cbb2-47cb-878b-ce9ee89b8d66-kube-api-access-pn529\") pod \"perf-node-gather-daemonset-nr62x\" (UID: \"9d404619-cbb2-47cb-878b-ce9ee89b8d66\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.331063 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.331041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:50.600196 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.600099    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"]
Sep 05 03:33:50.602977 ip-10-0-140-68 kubenswrapper[2568]: W0905 03:33:50.602951    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d404619_cbb2_47cb_878b_ce9ee89b8d66.slice/crio-758f0ff4b16934d010b920685871e1c9a06f9aa08196428c73d86a17c11bf62c WatchSource:0}: Error finding container 758f0ff4b16934d010b920685871e1c9a06f9aa08196428c73d86a17c11bf62c: Status 404 returned error can't find the container with id 758f0ff4b16934d010b920685871e1c9a06f9aa08196428c73d86a17c11bf62c
Sep 05 03:33:50.655272 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:50.655240    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-5n7gj_59de7810-3c4a-401f-b226-64f409289097/volume-data-source-validator/0.log"
Sep 05 03:33:51.416340 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.416308    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x" event={"ID":"9d404619-cbb2-47cb-878b-ce9ee89b8d66","Type":"ContainerStarted","Data":"93fe7aad80a0e28010a84b246bc8712a8a9ee0e0adde720e92ee8cc35129a6ca"}
Sep 05 03:33:51.416711 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.416356    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x" event={"ID":"9d404619-cbb2-47cb-878b-ce9ee89b8d66","Type":"ContainerStarted","Data":"758f0ff4b16934d010b920685871e1c9a06f9aa08196428c73d86a17c11bf62c"}
Sep 05 03:33:51.416711 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.416440    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:51.432445 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.432406    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x" podStartSLOduration=2.432391226 podStartE2EDuration="2.432391226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:33:51.431273372 +0000 UTC m=+4530.858389590" watchObservedRunningTime="2026-09-05 03:33:51.432391226 +0000 UTC m=+4530.859507451"
Sep 05 03:33:51.683817 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.683736    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhwx6_d7b1d500-6b4b-49e5-8f80-9ffde88c716c/dns/0.log"
Sep 05 03:33:51.707519 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.707489    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhwx6_d7b1d500-6b4b-49e5-8f80-9ffde88c716c/kube-rbac-proxy/0.log"
Sep 05 03:33:51.831907 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:51.831876    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5z4zs_02284cc8-068b-431e-a769-3ddd6faff502/dns-node-resolver/0.log"
Sep 05 03:33:52.432091 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:52.432040    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5fl8m_4e81178e-5034-4b0b-9009-8a964e5462ed/node-ca/0.log"
Sep 05 03:33:53.789047 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:53.789012    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tc4x8_91744050-b528-43d9-8bdb-8a2e67993ccb/discovery/0.log"
Sep 05 03:33:54.497790 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:54.497757    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5lgj_6078c5c9-bb6c-4f11-8ef2-a94064fdd4e3/serve-healthcheck-canary/0.log"
Sep 05 03:33:55.056499 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:55.056465    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvxmc_76bea6b1-457b-4002-9182-c09daa07cbf8/kube-rbac-proxy/0.log"
Sep 05 03:33:55.077766 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:55.077743    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvxmc_76bea6b1-457b-4002-9182-c09daa07cbf8/exporter/0.log"
Sep 05 03:33:55.098421 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:55.098386    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvxmc_76bea6b1-457b-4002-9182-c09daa07cbf8/extractor/0.log"
Sep 05 03:33:57.430171 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:57.430148    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-nr62x"
Sep 05 03:33:58.082364 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:33:58.082333    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54b595bc86-pq97q_50162386-1069-4362-8003-66dde904dfd8/manager/0.log"
Sep 05 03:34:01.733004 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:01.732908    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tc4x8_91744050-b528-43d9-8bdb-8a2e67993ccb/discovery/0.log"
Sep 05 03:34:03.310817 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:03.310781    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tc4x8_91744050-b528-43d9-8bdb-8a2e67993ccb/discovery/0.log"
Sep 05 03:34:06.260261 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:06.260231    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kptct_5f336f2e-51ff-4753-96d3-e42edb2b3dbc/migrator/0.log"
Sep 05 03:34:06.279023 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:06.278990    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kptct_5f336f2e-51ff-4753-96d3-e42edb2b3dbc/graceful-termination/0.log"
Sep 05 03:34:07.769977 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.769940    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:34:07.789810 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.789783    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/egress-router-binary-copy/0.log"
Sep 05 03:34:07.809702 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.809676    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/cni-plugins/0.log"
Sep 05 03:34:07.828445 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.828417    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/bond-cni-plugin/0.log"
Sep 05 03:34:07.848058 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.848031    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/routeoverride-cni/0.log"
Sep 05 03:34:07.866945 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.866926    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/whereabouts-cni-bincopy/0.log"
Sep 05 03:34:07.889224 ip-10-0-140-68 kubenswrapper[2568]: I0905 03:34:07.889200    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qsm2d_085eba4e-9a38-49ae-a3a9-e7aab3aee16b/whereabouts-cni/0.log"