Sep 05 02:16:11.136990 ip-10-0-143-8 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 05 02:16:11.137000 ip-10-0-143-8 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 05 02:16:11.137007 ip-10-0-143-8 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 05 02:16:11.137153 ip-10-0-143-8 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 05 02:16:21.386262 ip-10-0-143-8 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 05 02:16:21.386277 ip-10-0-143-8 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 98867cf28ddf465c9eb8487d62ceacd5 --
Sep 05 02:18:09.698843 ip-10-0-143-8 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:18:10.067885 ip-10-0-143-8 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:10.067885 ip-10-0-143-8 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:18:10.067885 ip-10-0-143-8 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:10.067885 ip-10-0-143-8 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:18:10.067885 ip-10-0-143-8 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:10.068527 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.067940    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:18:10.072023 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072007    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:10.072023 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072023    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072027    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072030    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072033    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072042    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072046    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072050    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072053    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072056    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072059    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072062    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072066    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072068    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072071    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072074    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072076    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072079    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072082    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072085    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:10.072092 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072087    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072090    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072093    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072096    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072099    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072102    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072105    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072108    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072110    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072113    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072116    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072119    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072121    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072124    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072127    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072131    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072136    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072140    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072142    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:10.072631 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072145    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072148    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072150    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072153    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072156    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072158    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072162    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072164    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072167    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072170    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072172    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072175    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072177    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072180    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072184    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072187    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072190    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072193    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072196    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:10.073076 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072199    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072201    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072204    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072207    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072209    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072212    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072214    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072217    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072220    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072222    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072225    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072227    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072238    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072241    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072243    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072246    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072249    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072253    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072256    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072259    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:10.073554 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072262    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072265    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072268    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072270    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072273    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072276    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072279    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072281    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072284    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072287    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072290    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072293    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072296    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072298    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072301    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072304    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072306    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072309    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072311    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072314    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:10.074053 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.072316    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:10.074822 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074810    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:10.074822 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074821    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074825    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074829    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074833    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074837    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074840    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074842    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074845    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074848    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074850    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074853    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074856    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074858    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074861    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074863    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074866    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074869    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074871    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074874    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074876    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:10.074916 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074879    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074882    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074884    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074887    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074890    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074892    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074895    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074897    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074900    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074902    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074905    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074907    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074910    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074912    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074916    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074920    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074925    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074928    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074931    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:10.075419 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074934    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074936    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074939    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074942    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074944    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074947    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074949    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074952    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074955    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074958    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074960    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074963    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074966    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074968    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074971    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074974    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074977    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074979    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074982    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:10.076137 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074984    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074988    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074991    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074994    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.074997    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075000    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075003    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075006    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075010    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075013    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075015    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075018    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075020    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075023    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075026    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075028    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075031    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075033    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075036    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:10.076924 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075038    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075041    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075043    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075046    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075048    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075051    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075054    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075056    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075058    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075061    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075064    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075066    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075069    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075072    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075074    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075076    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075079    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075082    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075085    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075088    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:10.077445 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.075092    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076289    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076310    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076315    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076320    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076325    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076330    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076334    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076339    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076347    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076354    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076360    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076371    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076375    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076380    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076385    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076389    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076395    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076406    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076411    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:10.077960 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076416    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076420    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076425    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076430    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076439    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076444    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076448    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076452    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076459    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076466    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076472    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076477    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076482    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076486    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076496    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076503    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076507    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076517    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:10.078441 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076522    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076527    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076532    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076537    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076545    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076550    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076554    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076573    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076579    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076587    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076591    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076601    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076605    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076610    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076615    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076641    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076801    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076809    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076812    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076816    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:10.078895 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076820    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076823    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076826    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076829    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076832    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076835    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076838    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076840    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076843    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076846    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076849    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076852    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076855    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076860    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076863    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076866    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076869    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076871    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076874    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076877    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:10.079365 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076880    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076882    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076885    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076887    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076890    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076893    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076896    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076898    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076902    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076905    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076908    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076911    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076913    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076916    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076919    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076921    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076924    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076927    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076930    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076933    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:10.079874 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076937    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.076940    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077015    2553 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077050    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077057    2553 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077062    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077066    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077069    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077073    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077078    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077081    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077084    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077087    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077091    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077094    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077097    2553 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077100    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077103    2553 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077106    2553 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077109    2553 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077114    2553 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077117    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077120    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077123    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:18:10.080399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077127    2553 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077131    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077134    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077137    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077141    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077143    2553 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077146    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077149    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077153    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077156    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077160    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077163    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077166    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077169    2553 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077172    2553 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077175    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077180    2553 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077183    2553 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077186    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077189    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077192    2553 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077195    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077198    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077201    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077204    2553 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:18:10.080988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077207    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077209    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077212    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077217    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077220    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077223    2553 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077226    2553 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077230    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077233    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077236    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077239    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077242    2553 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077245    2553 flags.go:64] FLAG: --help="false"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077248    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077251    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077254    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077257    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077260    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077264    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077266    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077269    2553 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077272    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077282    2553 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:18:10.081609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077286    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077289    2553 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077292    2553 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077295    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077298    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077301    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077303    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077306    2553 flags.go:64] FLAG: --lock-file=""
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077309    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077312    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077315    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077320    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077324    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077329    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077332    2553 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077335    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077339    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077342    2553 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077345    2553 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077349    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077352    2553 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077356    2553 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077359    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077362    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077365    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:18:10.082216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077368    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077371    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077374    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077377    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077384    2553 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077387    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077391    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077394    2553 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077397    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077402    2553 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077404    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077407    2553 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077410    2553 flags.go:64] FLAG: --port="10250"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077413    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077416    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b3e0d971b2af2291"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077419    2553 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077422    2553 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077426    2553 flags.go:64] FLAG: --register-node="true"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077429    2553 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077433    2553 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077436    2553 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077439    2553 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077442    2553 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077446    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077450    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:18:10.082855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077453    2553 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077456    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077459    2553 flags.go:64] FLAG: --runonce="false"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077462    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077465    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077468    2553 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077471    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077474    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077477    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077480    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077483    2553 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077486    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077489    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077492    2553 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077495    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077498    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077501    2553 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077504    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077510    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077513    2553 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077516    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077520    2553 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077523    2553 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077526    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077529    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:18:10.083469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077532    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077535    2553 flags.go:64] FLAG: --v="2"
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077539    2553 flags.go:64] FLAG: --version="false"
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077544    2553 flags.go:64] FLAG: --vmodule=""
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077548    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077552    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077677    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077682    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077685    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077688    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077691    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077693    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077696    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077698    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077703    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077706    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077708    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077711    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077714    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077716    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077719    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:10.084120 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077723    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077727    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077730    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077733    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077736    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077738    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077741    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077744    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077746    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077749    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077752    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077754    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077757    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077761    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077764    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077766    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077769    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077772    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077776    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077779    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:10.084680 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077782    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077785    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077787    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077790    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077792    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077796    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077799    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077802    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077805    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077807    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077810    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077813    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077815    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077818    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077821    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077824    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077827    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077829    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077832    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077834    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:10.085175 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077837    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077840    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077842    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077845    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077847    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077852    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077855    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077858    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077860    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077864    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077866    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077869    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077872    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077874    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077877    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077880    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077883    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077887    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077889    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:10.085686 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077892    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077895    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077899    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077902    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077905    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077908    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077911    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077914    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077917    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077920    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077923    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077926    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077929    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077932    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077935    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077938    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077941    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077944    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077948    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077950    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:10.086140 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077953    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077956    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077959    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077962    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.077966    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.077974    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.085696    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.085709    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.086595    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.086602    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086655    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086660    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:10.086657 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086664    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086668    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086671    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086674    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086676    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086679    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086682    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086686    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086690    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086694    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086697    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086700    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086703    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086706    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086709    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086712    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086715    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086718    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086721    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:10.086976 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086724    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086727    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086730    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086733    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086735    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086738    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086741    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086743    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086747    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086750    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086752    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086755    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086758    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086760    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086763    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086767    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086769    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086772    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086775    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086778    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:10.087467 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086780    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086783    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086786    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086788    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086791    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086794    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086797    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086799    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086802    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086804    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086807    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086810    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086812    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086815    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086817    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086820    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086822    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086825    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086828    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086830    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:10.088065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086833    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086836    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086839    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086842    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086844    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086847    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086850    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086854    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086856    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086859    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086861    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086864    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086866    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086869    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086872    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086874    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086878    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086881    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086884    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:10.088541 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086887    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086890    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086893    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086895    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086898    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086902    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086905    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086908    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086911    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086913    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086916    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086918    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086921    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086923    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086926    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086929    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086931    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086934    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:10.089006 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.086937    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.086942    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087054    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087059    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087062    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087065    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087068    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087071    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087074    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087077    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087080    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087083    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087085    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087088    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087091    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:10.089449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087093    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087096    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087099    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087101    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087104    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087106    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087109    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087111    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087114    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087116    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087119    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087122    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087124    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087128    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087132    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087135    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087138    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087141    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087159    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:10.089828 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087163    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087166    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087169    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087172    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087175    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087177    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087180    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087183    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087185    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087188    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087190    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087193    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087196    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087198    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087201    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087203    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087206    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087209    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087211    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087214    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:10.090345 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087216    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087219    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087222    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087224    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087227    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087229    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087232    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087234    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087237    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087240    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087243    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087245    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087248    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087251    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087253    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087257    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087261    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087263    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087266    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087269    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:10.090831 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087272    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087275    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087278    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087280    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087283    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087286    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087289    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087291    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087294    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087296    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087299    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087301    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087304    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087306    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087309    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087311    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087314    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087316    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087319    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087322    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:10.091360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087324    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087327    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087330    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087334    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087337    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087340    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:10.087342    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.087347    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.087452    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.089813    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.090908    2553 server.go:1019] "Starting client certificate rotation"
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.091023    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:18:10.091893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.091822    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:18:10.093783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.093767    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:18:10.099047 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.099030    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:18:10.113791 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.113763    2553 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:18:10.117543 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.117528    2553 log.go:25] "Validated CRI v1 image API"
Sep 05 02:18:10.118438 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.118420    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:18:10.123692 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.123675    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:10.125440 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.125424    2553 fs.go:135] Filesystem UUIDs: map[0795b00f-db9d-4d61-acac-460433b77602:/dev/nvme0n1p4 77da3908-7d59-4f79-82d1-3d9c2344d79a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 05 02:18:10.125494 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.125441    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:18:10.130522 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.130421    2553 manager.go:216] Machine: {Timestamp:2026-09-05 02:18:10.12995974 +0000 UTC m=+0.329322725 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094678 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263dc5a85842efdc5fbcca1d3bcd94 SystemUUID:ec263dc5-a858-42ef-dc5f-bcca1d3bcd94 BootID:98867cf2-8ddf-465c-9eb8-487d62ceacd5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3a:a7:c7:ed:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3a:a7:c7:ed:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:a5:3c:97:be:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:18:10.130522 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.130520    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:18:10.130637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.130606    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:18:10.132487 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.132469    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:18:10.132624 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.132489    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:18:10.133962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.133952    2553 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:18:10.133999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.133966    2553 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:18:10.133999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.133973    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:18:10.134822 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.134812    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:18:10.136696 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.136686    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:18:10.142181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142169    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:18:10.142296 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142287    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:18:10.142333 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142317    2553 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:18:10.142333 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142331    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:18:10.142388 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142354    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:18:10.142388 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142368    2553 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:18:10.142447 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142392    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:18:10.142725 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142713    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:18:10.142759 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.142736    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:18:10.145006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.144992    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:18:10.147989 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.147977    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:18:10.148028 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148006    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:18:10.148207 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148189    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:18:10.148297 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148215    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:18:10.148297 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148224    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:18:10.148297 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148230    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:18:10.148297 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148238    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:18:10.148297 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148244    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:18:10.148802 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148791    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:18:10.148802 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148802    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:18:10.148869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148808    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:18:10.148869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148819    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:18:10.148869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148827    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:18:10.148869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.148834    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:18:10.149612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.149602    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:18:10.149612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.149611    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:18:10.153434 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.153417    2553 server.go:1268] "Started kubelet"
Sep 05 02:18:10.153572 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.153505    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:18:10.153829 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.153616    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:18:10.154233 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.154024    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:18:10.154317 ip-10-0-143-8 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:18:10.155520 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.155503    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:18:10.155660 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.155579    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:18:10.158071 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.155733    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:18:10.158071 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.155866    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:18:10.158242 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.156988    2553 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:18:10.160158 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.160119    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:18:10.160732 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.159509    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-8.ec2.internal.18d24b3454d8ed28  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-8.ec2.internal,UID:ip-10-0-143-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-8.ec2.internal,},FirstTimestamp:2026-09-05 02:18:10.153393448 +0000 UTC m=+0.352756433,LastTimestamp:2026-09-05 02:18:10.153393448 +0000 UTC m=+0.352756433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-8.ec2.internal,}"
Sep 05 02:18:10.164695 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.164675    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:18:10.164776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.164678    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:10.164848 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.164829    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.165418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165401    2553 factory.go:55] Registering systemd factory
Sep 05 02:18:10.165418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165417    2553 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:18:10.165552 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165453    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:18:10.165552 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165459    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:18:10.165552 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165474    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:18:10.165713 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165604    2553 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:18:10.165713 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165612    2553 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:18:10.165713 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165692    2553 factory.go:153] Registering CRI-O factory
Sep 05 02:18:10.165713 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165711    2553 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:18:10.165884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165764    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:18:10.165884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165783    2553 factory.go:103] Registering Raw factory
Sep 05 02:18:10.165884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.165797    2553 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:18:10.166198 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.166184    2553 manager.go:349] Starting recovery of all containers
Sep 05 02:18:10.168352 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.168330    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 02:18:10.168455 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.168438    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 02:18:10.174636 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.174604    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2pn9n"
Sep 05 02:18:10.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.176404    2553 manager.go:354] Recovery completed
Sep 05 02:18:10.182569 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.182546    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.183032 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.183015    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2pn9n"
Sep 05 02:18:10.184068 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184051    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.184144 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184087    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.184144 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184099    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.184626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184611    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:18:10.184693 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184628    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:18:10.184693 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184619    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.184693 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.184646    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:18:10.185393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.185378    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.185450 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.185406    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.185450 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.185418    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.186149 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.186073    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-8.ec2.internal.18d24b3456ad0106  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-8.ec2.internal,UID:ip-10-0-143-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-8.ec2.internal,},FirstTimestamp:2026-09-05 02:18:10.184069382 +0000 UTC m=+0.383432369,LastTimestamp:2026-09-05 02:18:10.184069382 +0000 UTC m=+0.383432369,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-8.ec2.internal,}"
Sep 05 02:18:10.186535 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.186524    2553 policy_none.go:49] "None policy: Start"
Sep 05 02:18:10.186588 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.186539    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:18:10.186588 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.186549    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:18:10.188518 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.188505    2553 policy_none.go:47] "Start"
Sep 05 02:18:10.224206 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224191    2553 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.224233    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224244    2553 server.go:87] "Starting device plugin registration server"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224535    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224548    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224629    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224708    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.224717    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.225135    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.225177    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.240796    2553 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2590 into the system container \"/system.slice\": <nil>"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.325528    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.326439    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.326466    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.326480    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.326512    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.336158    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.337301 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.336179    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-8.ec2.internal\": node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.353213 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.353192    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.385011 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.384984    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:18:10.386357 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.386338    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:18:10.386438 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.386382    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:18:10.386438 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.386406    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:18:10.386511 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.386446    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:18:10.388909 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.388893    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:10.453926 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.453911    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.487416 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.487399    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"]
Sep 05 02:18:10.487475 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.487453    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.489513 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.489493    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.489591 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.489518    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.489591 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.489528    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.491835 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.491823    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.491949 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.491935    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.491992 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.491962    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.492495 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.492478    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.492495 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.492490    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.492639 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.492505    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.492639 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.492512    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.492639 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.492519    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.492639 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.492525    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.494810 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.494794    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.494860 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.494825    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:10.495527 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.495502    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:10.495606 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.495528    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:10.495606 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.495543    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:10.524078 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.524058    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-8.ec2.internal\" not found" node="ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.528284 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.528265    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-8.ec2.internal\" not found" node="ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.553971 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.553951    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.568818 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.568799    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51b31412bc99ad81f8dd5f9555a43fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal\" (UID: \"51b31412bc99ad81f8dd5f9555a43fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.568871 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.568829    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51b31412bc99ad81f8dd5f9555a43fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal\" (UID: \"51b31412bc99ad81f8dd5f9555a43fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.568871 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.568846    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e56369bb26fe910c3debbb11d5297fda-config\") pod \"kube-apiserver-proxy-ip-10-0-143-8.ec2.internal\" (UID: \"e56369bb26fe910c3debbb11d5297fda\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.654923 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.654871    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.669255 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.669237    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51b31412bc99ad81f8dd5f9555a43fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal\" (UID: \"51b31412bc99ad81f8dd5f9555a43fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.669312 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.669260    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51b31412bc99ad81f8dd5f9555a43fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal\" (UID: \"51b31412bc99ad81f8dd5f9555a43fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.669312 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.669284    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e56369bb26fe910c3debbb11d5297fda-config\") pod \"kube-apiserver-proxy-ip-10-0-143-8.ec2.internal\" (UID: \"e56369bb26fe910c3debbb11d5297fda\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.669379 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.669335    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51b31412bc99ad81f8dd5f9555a43fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal\" (UID: \"51b31412bc99ad81f8dd5f9555a43fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.669379 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.669349    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51b31412bc99ad81f8dd5f9555a43fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal\" (UID: \"51b31412bc99ad81f8dd5f9555a43fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.669446 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.669390    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e56369bb26fe910c3debbb11d5297fda-config\") pod \"kube-apiserver-proxy-ip-10-0-143-8.ec2.internal\" (UID: \"e56369bb26fe910c3debbb11d5297fda\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.755388 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.755371    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.826818 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.826801    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.831881 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:10.831629    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:10.855612 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.855592    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:10.956096 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:10.956040    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:11.056595 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:11.056555    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:11.090994 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.090975    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:18:11.091394 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.091151    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:18:11.156652 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:11.156631    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:11.185599 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.185551    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:13:10 +0000 UTC" deadline="2028-03-22 13:48:07.946332051 +0000 UTC"
Sep 05 02:18:11.185599 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.185593    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13547h29m56.760741498s"
Sep 05 02:18:11.257176 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:11.257152    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:11.356126 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.356100    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:11.358038 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:11.358017    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-8.ec2.internal\" not found"
Sep 05 02:18:11.421504 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:11.421475    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56369bb26fe910c3debbb11d5297fda.slice/crio-ecc9d3cba49f313b70f12a78b4d946705c41a967b9e411aa07fd3f3a4b2ef664 WatchSource:0}: Error finding container ecc9d3cba49f313b70f12a78b4d946705c41a967b9e411aa07fd3f3a4b2ef664: Status 404 returned error can't find the container with id ecc9d3cba49f313b70f12a78b4d946705c41a967b9e411aa07fd3f3a4b2ef664
Sep 05 02:18:11.424591 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.424562    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:11.426284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.426271    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:18:11.465186 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.465158    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:11.473780 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:11.473753    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b31412bc99ad81f8dd5f9555a43fb8.slice/crio-56ffd4e3c3533d8ac5cc848e116cf926158b5b2a3fc8521b29c5e831ee99c651 WatchSource:0}: Error finding container 56ffd4e3c3533d8ac5cc848e116cf926158b5b2a3fc8521b29c5e831ee99c651: Status 404 returned error can't find the container with id 56ffd4e3c3533d8ac5cc848e116cf926158b5b2a3fc8521b29c5e831ee99c651
Sep 05 02:18:11.481109 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.481094    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:11.482764 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.482747    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal"
Sep 05 02:18:11.492841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.492794    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:11.497233 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:11.497219    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:12.142901 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.142862    2553 apiserver.go:52] "Watching apiserver"
Sep 05 02:18:12.163791 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.163756    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:18:12.164498 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.164465    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-gc8cc","openshift-image-registry/node-ca-rsqww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal","openshift-multus/multus-f2sf6","openshift-multus/network-metrics-daemon-xmj5p","kube-system/global-pull-secret-syncer-2fzp6","openshift-dns/node-resolver-96gdl","openshift-multus/multus-additional-cni-plugins-zmrls","openshift-network-diagnostics/network-check-target-8vlxt","openshift-network-operator/iptables-alerter-h5dpn","openshift-ovn-kubernetes/ovnkube-node-996cw","kube-system/konnectivity-agent-wxqrh","kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"]
Sep 05 02:18:12.165874 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.165850    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:12.167391 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.167371    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.169910 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.169888    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.170836 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.170815    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qbb6b\""
Sep 05 02:18:12.170955 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.170936    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.171071 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.171052    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.172155 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.172136    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.172884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.172861    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.173094 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.173074    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:18:12.173475 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.173459    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6hvn\""
Sep 05 02:18:12.173851 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.173829    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.174464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.174445    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.174960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.174936    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zqlfn\""
Sep 05 02:18:12.176103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176074    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-var-lib-kubelet\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysconfig\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176138    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysctl-d\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176163    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysctl-conf\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176188    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-lib-modules\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176211    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-host\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176234    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/952593be-283f-4925-b73b-d5a62c85b6ec-etc-tuned\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176257    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-sys\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176282    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-modprobe-d\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176308    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-kubernetes\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176333    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-systemd\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176354    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-run\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176376    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/952593be-283f-4925-b73b-d5a62c85b6ec-tmp\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176402    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhxn\" (UniqueName: \"kubernetes.io/projected/952593be-283f-4925-b73b-d5a62c85b6ec-kube-api-access-gbhxn\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.176894 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.176812    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.177459 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.177438    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:18:12.177779 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.177762    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:18:12.177875 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.177762    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:18:12.177933 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.177879    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.178252 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.178228    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.178451 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.178430    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.178528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.178440    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:18:12.179119 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.179096    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.179236 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.179185    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.181694    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.181749    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.181942    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.182000    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.182002    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.181942    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.182043    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tmr87\""
Sep 05 02:18:12.182567 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.182305    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-46kfj\""
Sep 05 02:18:12.185174 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.185152    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.186286 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.186264    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:12.186392 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.186344    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:12.187756 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.187737    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gmbh2\""
Sep 05 02:18:12.187853 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.187736    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:18:12.191676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.188861    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.192458 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.192422    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.192784 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.192763    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.192872 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.192815    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.194012 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.193989    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6wrt7\""
Sep 05 02:18:12.194246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.194229    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:18:12.194833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.194807    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.195442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.195424    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:18:12.195653 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.195632    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:18:12.196799 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.196778    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:18:12.197176 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.197156    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:18:12.197304 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.197284    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:18:12.197394 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.197295    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7ndj9\""
Sep 05 02:18:12.197720 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.197698    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:18:12.197965 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.197947    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:18:12.198051 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.197984    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:18:12.198051 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.198044    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m4tnv\""
Sep 05 02:18:12.216258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.216238    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjfwq"
Sep 05 02:18:12.216258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.216259    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjfwq"
Sep 05 02:18:12.267319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.267301    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:18:12.276662 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276637    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-ovnkube-script-lib\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.276758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276680    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysconfig\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.276758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276708    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysctl-d\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.276758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276735    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjgx9\" (UniqueName: \"kubernetes.io/projected/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-kube-api-access-cjgx9\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276760    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-lib-modules\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276777    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4059ab2-be92-4752-8b64-e8b5044bfbff-konnectivity-ca\") pod \"konnectivity-agent-wxqrh\" (UID: \"c4059ab2-be92-4752-8b64-e8b5044bfbff\") " pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276791    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-serviceca\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276806    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-cnibin\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276821    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-kubelet\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276836    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-cni-binary-copy\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276853    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-kubelet-config\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276868    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-sys\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276883    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzgv\" (UniqueName: \"kubernetes.io/projected/687cd57d-c36b-4913-b647-39fcfe11fc70-kube-api-access-6jzgv\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276903    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d380942-90b8-482d-a917-d4d3435fb445-hosts-file\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276918    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpcg\" (UniqueName: \"kubernetes.io/projected/7feca731-5171-453d-b2c6-9a5115d03b92-kube-api-access-jxpcg\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.276928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276932    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-hostroot\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276946    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-etc-selinux\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276966    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-system-cni-dir\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.276987    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-etc-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277013    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-conf-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277034    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277066    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-registration-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277087    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277121    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-dbus\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277144    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-node-log\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277168    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-socket-dir-parent\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277185    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-daemon-config\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277199    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-log-socket\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277215    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-kubernetes\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277229    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-host\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.277509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277243    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-multus-certs\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277259    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gc5\" (UniqueName: \"kubernetes.io/projected/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-kube-api-access-j4gc5\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277274    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-systemd-units\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277314    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-env-overrides\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277333    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4059ab2-be92-4752-8b64-e8b5044bfbff-agent-certs\") pod \"konnectivity-agent-wxqrh\" (UID: \"c4059ab2-be92-4752-8b64-e8b5044bfbff\") " pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277351    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d380942-90b8-482d-a917-d4d3435fb445-tmp-dir\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-k8s-cni-cncf-io\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277382    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-etc-kubernetes\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277401    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-var-lib-kubelet\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277423    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0437b2bf-4237-4912-bc21-8dd6e444cfda-cni-binary-copy\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277449    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysctl-conf\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277473    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/952593be-283f-4925-b73b-d5a62c85b6ec-etc-tuned\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277497    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-slash\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277515    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277531    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-netns\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277550    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-systemd\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-run\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278227 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277600    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277631    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-iptables-alerter-script\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277678    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-host-slash\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277704    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-socket-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277724    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysconfig\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277885    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysctl-d\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277907    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-kubernetes\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.277734    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278212    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-lib-modules\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278278    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-var-lib-kubelet\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278319    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-sys\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278414    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-sysctl-conf\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278529    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-run\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278576    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-systemd\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278585    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhxn\" (UniqueName: \"kubernetes.io/projected/952593be-283f-4925-b73b-d5a62c85b6ec-kube-api-access-gbhxn\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278623    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-systemd\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278655    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-var-lib-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.278962 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-ovn\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278708    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-cni-netd\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278734    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-cni-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278761    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-device-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278787    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-cnibin\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278811    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-ovnkube-config\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278838    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-cni-multus\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278883    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/d6862640-b4c3-47c5-a10b-13d7df1a9948-kube-api-access-md2sb\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278922    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-host\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.278979    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-run-netns\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279055    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-system-cni-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279058    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-host\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279098    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-cni-bin\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279125    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-sys-fs\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279159    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279186    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.279650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279239    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-modprobe-d\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279286    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-kubelet\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279316    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-cni-bin\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279388    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/687cd57d-c36b-4913-b647-39fcfe11fc70-ovn-node-metrics-cert\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279400    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/952593be-283f-4925-b73b-d5a62c85b6ec-etc-modprobe-d\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279418    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfnj\" (UniqueName: \"kubernetes.io/projected/0d380942-90b8-482d-a917-d4d3435fb445-kube-api-access-plfnj\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279446    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rvp\" (UniqueName: \"kubernetes.io/projected/0437b2bf-4237-4912-bc21-8dd6e444cfda-kube-api-access-g6rvp\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279483    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkhk\" (UniqueName: \"kubernetes.io/projected/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-kube-api-access-6zkhk\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279540    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/952593be-283f-4925-b73b-d5a62c85b6ec-tmp\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279566    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.280418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.279646    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:18:12.285354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.285175    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/952593be-283f-4925-b73b-d5a62c85b6ec-etc-tuned\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.287712 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.287689    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/952593be-283f-4925-b73b-d5a62c85b6ec-tmp\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.289802 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.289779    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhxn\" (UniqueName: \"kubernetes.io/projected/952593be-283f-4925-b73b-d5a62c85b6ec-kube-api-access-gbhxn\") pod \"tuned-gc8cc\" (UID: \"952593be-283f-4925-b73b-d5a62c85b6ec\") " pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.380496 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380465    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-log-socket\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.380626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380509    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-host\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.380626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380538    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-multus-certs\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.380626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380571    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gc5\" (UniqueName: \"kubernetes.io/projected/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-kube-api-access-j4gc5\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.380626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380615    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-systemd-units\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380640    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-env-overrides\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380664    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4059ab2-be92-4752-8b64-e8b5044bfbff-agent-certs\") pod \"konnectivity-agent-wxqrh\" (UID: \"c4059ab2-be92-4752-8b64-e8b5044bfbff\") " pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380691    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d380942-90b8-482d-a917-d4d3435fb445-tmp-dir\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380716    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-k8s-cni-cncf-io\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380742    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-etc-kubernetes\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380767    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0437b2bf-4237-4912-bc21-8dd6e444cfda-cni-binary-copy\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380793    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-slash\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380819    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380844    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-netns\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.380883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380872    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380898    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-iptables-alerter-script\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380923    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-host-slash\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380948    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-socket-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.380977    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381003    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-systemd\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-systemd\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-etc-kubernetes\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381081    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-var-lib-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381112    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-ovn\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381126    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-slash\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381145    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-cni-netd\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381161    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381190    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-cni-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381204    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-netns\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381217    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-device-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381243    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.381382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381243    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-cnibin\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381278    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-cnibin\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381281    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-ovnkube-config\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381314    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-cni-multus\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381342    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/d6862640-b4c3-47c5-a10b-13d7df1a9948-kube-api-access-md2sb\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-run-netns\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381395    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-system-cni-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381417    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-cni-bin\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381428    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-host-slash\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381442    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-sys-fs\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.381924    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-ovnkube-config\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382027    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0437b2bf-4237-4912-bc21-8dd6e444cfda-cni-binary-copy\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382036    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382068    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-kubelet\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382077    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-var-lib-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382094    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-cni-bin\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.382245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382136    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/687cd57d-c36b-4913-b647-39fcfe11fc70-ovn-node-metrics-cert\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382164    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnj\" (UniqueName: \"kubernetes.io/projected/0d380942-90b8-482d-a917-d4d3435fb445-kube-api-access-plfnj\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382192    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rvp\" (UniqueName: \"kubernetes.io/projected/0437b2bf-4237-4912-bc21-8dd6e444cfda-kube-api-access-g6rvp\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382216    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382249    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkhk\" (UniqueName: \"kubernetes.io/projected/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-kube-api-access-6zkhk\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382277    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382305    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-ovnkube-script-lib\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382333    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjgx9\" (UniqueName: \"kubernetes.io/projected/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-kube-api-access-cjgx9\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4059ab2-be92-4752-8b64-e8b5044bfbff-konnectivity-ca\") pod \"konnectivity-agent-wxqrh\" (UID: \"c4059ab2-be92-4752-8b64-e8b5044bfbff\") " pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-serviceca\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382418    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-cnibin\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382443    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-kubelet\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382474    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-cni-binary-copy\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382591    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382618    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-iptables-alerter-script\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382663    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-kubelet-config\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382671    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-log-socket\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382698    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzgv\" (UniqueName: \"kubernetes.io/projected/687cd57d-c36b-4913-b647-39fcfe11fc70-kube-api-access-6jzgv\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382725    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d380942-90b8-482d-a917-d4d3435fb445-hosts-file\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382713    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-host\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.382736    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382754    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpcg\" (UniqueName: \"kubernetes.io/projected/7feca731-5171-453d-b2c6-9a5115d03b92-kube-api-access-jxpcg\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-hostroot\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.382810    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:12.882787526 +0000 UTC m=+3.082150504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382823    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-hostroot\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-etc-selinux\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382862    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-kubelet\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382877    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-system-cni-dir\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382898    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-cni-bin\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382906    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-etc-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382935    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-conf-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382962    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.382989    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-registration-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383020    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.383960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383051    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383077    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-dbus\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383103    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-node-log\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383130    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-socket-dir-parent\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383158    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-daemon-config\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383161    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-cni-binary-copy\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383215    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-cni-multus\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383264    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-multus-certs\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383368    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-run-ovn\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383404    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-cni-netd\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383527    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-cni-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383532    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-systemd-units\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383573    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-device-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383660    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-run-netns\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383707    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-sys-fs\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.383742    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-cni-bin\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384149    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6862640-b4c3-47c5-a10b-13d7df1a9948-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384209    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-node-log\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.384602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384225    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-daemon-config\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384262    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-dbus\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384287    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-system-cni-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384348    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-socket-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384388    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-socket-dir-parent\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384505    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.384597    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.384679    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:12.884663467 +0000 UTC m=+3.084026446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384731    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-var-lib-kubelet\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384817    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-system-cni-dir\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.384945    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/687cd57d-c36b-4913-b647-39fcfe11fc70-etc-openvswitch\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385005    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-multus-conf-dir\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385051    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385055    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-ovnkube-script-lib\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385099    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-registration-dir\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385110    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-kubelet-config\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385242    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6862640-b4c3-47c5-a10b-13d7df1a9948-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.385491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385253    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d380942-90b8-482d-a917-d4d3435fb445-hosts-file\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385357    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-cnibin\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385409    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/687cd57d-c36b-4913-b647-39fcfe11fc70-env-overrides\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385456    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-etc-selinux\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385465    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0437b2bf-4237-4912-bc21-8dd6e444cfda-host-run-k8s-cni-cncf-io\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385519    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4059ab2-be92-4752-8b64-e8b5044bfbff-konnectivity-ca\") pod \"konnectivity-agent-wxqrh\" (UID: \"c4059ab2-be92-4752-8b64-e8b5044bfbff\") " pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385735    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-serviceca\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385774    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d380942-90b8-482d-a917-d4d3435fb445-tmp-dir\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.386349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.385824    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/687cd57d-c36b-4913-b647-39fcfe11fc70-ovn-node-metrics-cert\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.388739 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.388717    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4059ab2-be92-4752-8b64-e8b5044bfbff-agent-certs\") pod \"konnectivity-agent-wxqrh\" (UID: \"c4059ab2-be92-4752-8b64-e8b5044bfbff\") " pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.391791 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.391652    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal" event={"ID":"51b31412bc99ad81f8dd5f9555a43fb8","Type":"ContainerStarted","Data":"56ffd4e3c3533d8ac5cc848e116cf926158b5b2a3fc8521b29c5e831ee99c651"}
Sep 05 02:18:12.393943 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.393767    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal" event={"ID":"e56369bb26fe910c3debbb11d5297fda","Type":"ContainerStarted","Data":"ecc9d3cba49f313b70f12a78b4d946705c41a967b9e411aa07fd3f3a4b2ef664"}
Sep 05 02:18:12.403217 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.403185    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfnj\" (UniqueName: \"kubernetes.io/projected/0d380942-90b8-482d-a917-d4d3435fb445-kube-api-access-plfnj\") pod \"node-resolver-96gdl\" (UID: \"0d380942-90b8-482d-a917-d4d3435fb445\") " pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.403394 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.403362    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:12.403394 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.403389    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:12.403578 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.403403    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:12.403578 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.403458    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:12.903442702 +0000 UTC m=+3.102805680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:12.408142 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.408085    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjgx9\" (UniqueName: \"kubernetes.io/projected/a2a60161-98fa-4b9d-87a1-457e09a1b7fa-kube-api-access-cjgx9\") pod \"node-ca-rsqww\" (UID: \"a2a60161-98fa-4b9d-87a1-457e09a1b7fa\") " pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.408142 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.408099    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gc5\" (UniqueName: \"kubernetes.io/projected/ad4dd761-a3cd-4bb3-8db9-62bc2388ad89-kube-api-access-j4gc5\") pod \"aws-ebs-csi-driver-node-mpdlc\" (UID: \"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.410849 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.409960    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkhk\" (UniqueName: \"kubernetes.io/projected/5c17b789-5a0b-4ebc-9dda-0ebca96bc357-kube-api-access-6zkhk\") pod \"iptables-alerter-h5dpn\" (UID: \"5c17b789-5a0b-4ebc-9dda-0ebca96bc357\") " pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.410849 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.410011    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzgv\" (UniqueName: \"kubernetes.io/projected/687cd57d-c36b-4913-b647-39fcfe11fc70-kube-api-access-6jzgv\") pod \"ovnkube-node-996cw\" (UID: \"687cd57d-c36b-4913-b647-39fcfe11fc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.411026 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.410980    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rvp\" (UniqueName: \"kubernetes.io/projected/0437b2bf-4237-4912-bc21-8dd6e444cfda-kube-api-access-g6rvp\") pod \"multus-f2sf6\" (UID: \"0437b2bf-4237-4912-bc21-8dd6e444cfda\") " pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.411499 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.411472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/d6862640-b4c3-47c5-a10b-13d7df1a9948-kube-api-access-md2sb\") pod \"multus-additional-cni-plugins-zmrls\" (UID: \"d6862640-b4c3-47c5-a10b-13d7df1a9948\") " pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.413115 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.413089    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpcg\" (UniqueName: \"kubernetes.io/projected/7feca731-5171-453d-b2c6-9a5115d03b92-kube-api-access-jxpcg\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.487211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.487180    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gc8cc"
Sep 05 02:18:12.495337 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.495310    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc"
Sep 05 02:18:12.504169 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.504131    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rsqww"
Sep 05 02:18:12.515946 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.515925    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zmrls"
Sep 05 02:18:12.525745 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.525726    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-96gdl"
Sep 05 02:18:12.530819 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.530802    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:12.533963 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.533946    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2sf6"
Sep 05 02:18:12.544743 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.544717    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h5dpn"
Sep 05 02:18:12.553694 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.553675    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:12.560474 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.560452    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:12.887245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.887167    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:12.887245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.887229    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:12.887460 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.887339    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:12.887460 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.887399    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.8873811 +0000 UTC m=+4.086744081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:12.887460 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.887340    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:12.887648 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.887512    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.887490737 +0000 UTC m=+4.086853724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:12.988001 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:12.987962    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:12.988170 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.988105    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:12.988170 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.988124    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:12.988170 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.988137    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:12.988325 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:12.988220    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.988199154 +0000 UTC m=+4.187562130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:13.221243 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.221161    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:12 +0000 UTC" deadline="2028-03-31 19:41:33.036894129 +0000 UTC"
Sep 05 02:18:13.221243 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.221191    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13769h23m19.815706894s"
Sep 05 02:18:13.387527 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.387497    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:13.387734 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.387630    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:13.387734 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.387692    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:13.387860 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.387801    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:13.563913 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.563879    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d380942_90b8_482d_a917_d4d3435fb445.slice/crio-4829c536a0c0467a573bc8022c3dfd786fe42e44bdd27c2317c3a5178e3b7959 WatchSource:0}: Error finding container 4829c536a0c0467a573bc8022c3dfd786fe42e44bdd27c2317c3a5178e3b7959: Status 404 returned error can't find the container with id 4829c536a0c0467a573bc8022c3dfd786fe42e44bdd27c2317c3a5178e3b7959
Sep 05 02:18:13.565515 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.565486    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687cd57d_c36b_4913_b647_39fcfe11fc70.slice/crio-17135ca37859ea514a5420af76bee0234566b57c89804af21652a5b5bda2520b WatchSource:0}: Error finding container 17135ca37859ea514a5420af76bee0234566b57c89804af21652a5b5bda2520b: Status 404 returned error can't find the container with id 17135ca37859ea514a5420af76bee0234566b57c89804af21652a5b5bda2520b
Sep 05 02:18:13.566360 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.566330    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952593be_283f_4925_b73b_d5a62c85b6ec.slice/crio-1b8bedad4294e076acbb5faccc695ff8f8bafef08bc2bea5255d1e0f8cc67e49 WatchSource:0}: Error finding container 1b8bedad4294e076acbb5faccc695ff8f8bafef08bc2bea5255d1e0f8cc67e49: Status 404 returned error can't find the container with id 1b8bedad4294e076acbb5faccc695ff8f8bafef08bc2bea5255d1e0f8cc67e49
Sep 05 02:18:13.567666 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.567220    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4dd761_a3cd_4bb3_8db9_62bc2388ad89.slice/crio-98cada44937de58aa3bf517195a0bd4736923a5dfc13c956a12458b7a78a9743 WatchSource:0}: Error finding container 98cada44937de58aa3bf517195a0bd4736923a5dfc13c956a12458b7a78a9743: Status 404 returned error can't find the container with id 98cada44937de58aa3bf517195a0bd4736923a5dfc13c956a12458b7a78a9743
Sep 05 02:18:13.570471 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.570448    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c17b789_5a0b_4ebc_9dda_0ebca96bc357.slice/crio-c4a75f26604a9eca89a6561eb3708101e0f36989faa0d54432f1f04b410e5719 WatchSource:0}: Error finding container c4a75f26604a9eca89a6561eb3708101e0f36989faa0d54432f1f04b410e5719: Status 404 returned error can't find the container with id c4a75f26604a9eca89a6561eb3708101e0f36989faa0d54432f1f04b410e5719
Sep 05 02:18:13.571950 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.571834    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0437b2bf_4237_4912_bc21_8dd6e444cfda.slice/crio-0a51cf6a4361b7f8fc81ff35f5a76c319c6eda9069b80df5a15992f47315f886 WatchSource:0}: Error finding container 0a51cf6a4361b7f8fc81ff35f5a76c319c6eda9069b80df5a15992f47315f886: Status 404 returned error can't find the container with id 0a51cf6a4361b7f8fc81ff35f5a76c319c6eda9069b80df5a15992f47315f886
Sep 05 02:18:13.573169 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.573060    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a60161_98fa_4b9d_87a1_457e09a1b7fa.slice/crio-b8c45d41a624158f4beb1cdf8c7282384dbe4bcf9bc5afa34aca6fe740bd0abc WatchSource:0}: Error finding container b8c45d41a624158f4beb1cdf8c7282384dbe4bcf9bc5afa34aca6fe740bd0abc: Status 404 returned error can't find the container with id b8c45d41a624158f4beb1cdf8c7282384dbe4bcf9bc5afa34aca6fe740bd0abc
Sep 05 02:18:13.574827 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.574798    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4059ab2_be92_4752_8b64_e8b5044bfbff.slice/crio-29fc77291e78207c4b205ad389c811e0bf66c02d13f8ceb8c2677c9c00be3a34 WatchSource:0}: Error finding container 29fc77291e78207c4b205ad389c811e0bf66c02d13f8ceb8c2677c9c00be3a34: Status 404 returned error can't find the container with id 29fc77291e78207c4b205ad389c811e0bf66c02d13f8ceb8c2677c9c00be3a34
Sep 05 02:18:13.576807 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:13.575894    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6862640_b4c3_47c5_a10b_13d7df1a9948.slice/crio-ba988559476fc201c0026c46361e13c4f2bf8a8cf3bf95ec54f30835540af119 WatchSource:0}: Error finding container ba988559476fc201c0026c46361e13c4f2bf8a8cf3bf95ec54f30835540af119: Status 404 returned error can't find the container with id ba988559476fc201c0026c46361e13c4f2bf8a8cf3bf95ec54f30835540af119
Sep 05 02:18:13.897456 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.897423    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:13.897556 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.897477    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:13.897628 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.897612    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:13.897680 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.897640    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:13.897718 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.897692    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:15.897662748 +0000 UTC m=+6.097025735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:13.897718 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.897707    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:15.897700704 +0000 UTC m=+6.097063677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:13.998331 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:13.998297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:13.998532 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.998490    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:13.998532 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.998512    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:13.998532 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.998522    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:13.998723 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:13.998598    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:15.998564638 +0000 UTC m=+6.197927626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:14.221947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.221825    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:12 +0000 UTC" deadline="2028-03-21 08:14:11.65126971 +0000 UTC"
Sep 05 02:18:14.221947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.221877    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13517h55m57.42939636s"
Sep 05 02:18:14.389247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.389210    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:14.389425 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:14.389355    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:14.408953 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.408910    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal" event={"ID":"e56369bb26fe910c3debbb11d5297fda","Type":"ContainerStarted","Data":"fe14c587e06a82aca5e184918f4c071414bb871fdb05574187da8059efa52500"}
Sep 05 02:18:14.414820 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.414655    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2sf6" event={"ID":"0437b2bf-4237-4912-bc21-8dd6e444cfda","Type":"ContainerStarted","Data":"0a51cf6a4361b7f8fc81ff35f5a76c319c6eda9069b80df5a15992f47315f886"}
Sep 05 02:18:14.427561 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.427489    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerStarted","Data":"ba988559476fc201c0026c46361e13c4f2bf8a8cf3bf95ec54f30835540af119"}
Sep 05 02:18:14.431204 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.431173    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rsqww" event={"ID":"a2a60161-98fa-4b9d-87a1-457e09a1b7fa","Type":"ContainerStarted","Data":"b8c45d41a624158f4beb1cdf8c7282384dbe4bcf9bc5afa34aca6fe740bd0abc"}
Sep 05 02:18:14.439260 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.439225    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc" event={"ID":"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89","Type":"ContainerStarted","Data":"98cada44937de58aa3bf517195a0bd4736923a5dfc13c956a12458b7a78a9743"}
Sep 05 02:18:14.455322 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.455289    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gc8cc" event={"ID":"952593be-283f-4925-b73b-d5a62c85b6ec","Type":"ContainerStarted","Data":"1b8bedad4294e076acbb5faccc695ff8f8bafef08bc2bea5255d1e0f8cc67e49"}
Sep 05 02:18:14.460348 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.460315    2553 generic.go:356] "Generic (PLEG): container finished" podID="51b31412bc99ad81f8dd5f9555a43fb8" containerID="caa9e44fffa3bdd42e20e6581ae0db5dc97e708f167e2e97cbaad86ebffc09d0" exitCode=0
Sep 05 02:18:14.460482 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.460403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal" event={"ID":"51b31412bc99ad81f8dd5f9555a43fb8","Type":"ContainerDied","Data":"caa9e44fffa3bdd42e20e6581ae0db5dc97e708f167e2e97cbaad86ebffc09d0"}
Sep 05 02:18:14.469171 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.469140    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wxqrh" event={"ID":"c4059ab2-be92-4752-8b64-e8b5044bfbff","Type":"ContainerStarted","Data":"29fc77291e78207c4b205ad389c811e0bf66c02d13f8ceb8c2677c9c00be3a34"}
Sep 05 02:18:14.474118 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.474052    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h5dpn" event={"ID":"5c17b789-5a0b-4ebc-9dda-0ebca96bc357","Type":"ContainerStarted","Data":"c4a75f26604a9eca89a6561eb3708101e0f36989faa0d54432f1f04b410e5719"}
Sep 05 02:18:14.478489 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.478461    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"17135ca37859ea514a5420af76bee0234566b57c89804af21652a5b5bda2520b"}
Sep 05 02:18:14.479947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.479881    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-8.ec2.internal" podStartSLOduration=3.479861205 podStartE2EDuration="3.479861205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:14.425572954 +0000 UTC m=+4.624935953" watchObservedRunningTime="2026-09-05 02:18:14.479861205 +0000 UTC m=+4.679224203"
Sep 05 02:18:14.503046 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:14.502942    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-96gdl" event={"ID":"0d380942-90b8-482d-a917-d4d3435fb445","Type":"ContainerStarted","Data":"4829c536a0c0467a573bc8022c3dfd786fe42e44bdd27c2317c3a5178e3b7959"}
Sep 05 02:18:15.387579 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:15.387077    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:15.387579 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:15.387116    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:15.387579 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:15.387223    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:15.387579 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:15.387367    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:15.557670 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:15.554230    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal" event={"ID":"51b31412bc99ad81f8dd5f9555a43fb8","Type":"ContainerStarted","Data":"c4518de31f5c0581d001cff0fd682850aa0aa325b158eae7e1741e5e96a558da"}
Sep 05 02:18:15.915419 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:15.915378    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:15.915603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:15.915464    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:15.915710 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:15.915627    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:15.915710 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:15.915690    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.915671845 +0000 UTC m=+10.115034833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:15.916103 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:15.916084    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:15.916193 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:15.916171    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.916154684 +0000 UTC m=+10.115517683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:16.016117 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:16.016075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:16.016285 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:16.016256    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:16.016285 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:16.016276    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:16.016408 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:16.016290    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:16.016408 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:16.016352    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.01633347 +0000 UTC m=+10.215696447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:16.388777 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:16.388277    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:16.388777 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:16.388371    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:17.387246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:17.386725    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:17.387246 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:17.386879    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:17.387246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:17.387173    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:17.387567 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:17.387269    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:18.387099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:18.387069    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:18.387534 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:18.387198    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:19.387947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:19.387419    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:19.387947 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:19.387813    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:19.387947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:19.387491    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:19.388483 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:19.388053    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:19.947782 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:19.947497    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:19.947782 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:19.947599    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:19.947782 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:19.947679    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:19.947782 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:19.947729    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:19.947782 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:19.947751    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:27.947732703 +0000 UTC m=+18.147095680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:19.947782 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:19.947787    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:27.947770967 +0000 UTC m=+18.147133939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:20.050741 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:20.048312    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:20.056827 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:20.056800    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:20.056944 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:20.056834    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:20.056944 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:20.056849    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:20.056944 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:20.056917    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:28.05689878 +0000 UTC m=+18.256261762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:20.388390 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:20.387872    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:20.388390 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:20.387979    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:21.386759 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:21.386728    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:21.386940 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:21.386851    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:21.387283 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:21.387242    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:21.387421 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:21.387354    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:22.387679 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:22.387633    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:22.388093 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:22.387765    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:23.386600 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:23.386569    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:23.386757 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:23.386570    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:23.386757 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:23.386671    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:23.386757 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:23.386739    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:24.386821 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:24.386795    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:24.387191 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:24.386899    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:25.386748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:25.386714    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:25.386950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:25.386723    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:25.386950 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:25.386841    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:25.386950 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:25.386927    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:26.390146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:26.390114    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:26.390628 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:26.390222    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:27.386860 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:27.386823    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:27.387076 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:27.386954    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:27.387076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:27.386965    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:27.387076 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:27.387050    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:28.005997 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:28.005964    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:28.006440 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:28.006038    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:28.006440 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.006134    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:28.006440 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.006144    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:28.006440 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.006210    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:44.006191102 +0000 UTC m=+34.205554077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:28.006440 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.006233    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:44.006221612 +0000 UTC m=+34.205584591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:28.107004 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:28.106971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:28.107155 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.107117    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:28.107155 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.107135    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:28.107155 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.107148    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:28.107292 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.107203    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:44.10718592 +0000 UTC m=+34.306548898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:28.386908 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:28.386811    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:28.387099 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:28.386946    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:29.387002 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:29.386969    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:29.387456 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:29.386969    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:29.387456 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:29.387091    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:29.387456 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:29.387156    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:30.387854 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:30.387816    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:30.388305 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:30.387934    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:31.386820 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:31.386788    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:31.386999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:31.386800    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:31.386999 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:31.386917    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:31.386999 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:31.386970    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:32.386927 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.386655    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:32.387601 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:32.386963    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:32.591435 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.591401    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wxqrh" event={"ID":"c4059ab2-be92-4752-8b64-e8b5044bfbff","Type":"ContainerStarted","Data":"d66787c9f8e68e6e7fd0b91376275f6fe8cf5f4b41299d93a7ac5933c090155b"}
Sep 05 02:18:32.594729 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.594686    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"8e5638822f6d25b818e82dda90f7821a7b29ef369c996868ffdfbeab0e43638c"}
Sep 05 02:18:32.594905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.594796    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"6a1b719ca38f0d60b47dbd351c2db3bbfe5da352f8361f6af7830ba9ec9575b3"}
Sep 05 02:18:32.594905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.594846    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"327ca3cd8a4a673a5a154852aec9222875c46c372cfc639d1821199dab6de087"}
Sep 05 02:18:32.594905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.594902    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"b5bd05e8790c3d1e6d55f891156cbd67c41d2afbbcbfbb6d0f87d80dfc6a469a"}
Sep 05 02:18:32.595139 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.594947    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"3497240f56f4525c247426cb63b4b3202e5d2693aebb89fd916fa9a17ca01e0c"}
Sep 05 02:18:32.595139 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.594995    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"9080393fc45f44d6d2495ecc84ba28efe2e3eccde67d6e7da027fb742328cc47"}
Sep 05 02:18:32.596686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.596654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-96gdl" event={"ID":"0d380942-90b8-482d-a917-d4d3435fb445","Type":"ContainerStarted","Data":"ae0261f7c5eeedb398bbaf38bbd9fe7c4649165444963db7cdd86d610a4c6a50"}
Sep 05 02:18:32.598399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.598363    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2sf6" event={"ID":"0437b2bf-4237-4912-bc21-8dd6e444cfda","Type":"ContainerStarted","Data":"da04c592618c7f87c743dd70bfc142343048e3972d13c5ba2e8e17a9cb0f6bbe"}
Sep 05 02:18:32.600337 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.600307    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6862640-b4c3-47c5-a10b-13d7df1a9948" containerID="7109f45567343efe64851d9c35acecd7477d7ce311e5dd3d4ca8e0c537a559d5" exitCode=0
Sep 05 02:18:32.600442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.600408    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerDied","Data":"7109f45567343efe64851d9c35acecd7477d7ce311e5dd3d4ca8e0c537a559d5"}
Sep 05 02:18:32.602300 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.602279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rsqww" event={"ID":"a2a60161-98fa-4b9d-87a1-457e09a1b7fa","Type":"ContainerStarted","Data":"112fe7e3947bde815a359604881f6ddcef2933849d271609e51829fc8b568ac4"}
Sep 05 02:18:32.604290 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.604267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc" event={"ID":"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89","Type":"ContainerStarted","Data":"70eab99e35230cc9ede871255cf76501350b88cd2b2cde3d2cd099acf9094107"}
Sep 05 02:18:32.605923 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.605902    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gc8cc" event={"ID":"952593be-283f-4925-b73b-d5a62c85b6ec","Type":"ContainerStarted","Data":"774c81d0f04efe75eb7be00668cf93821e1ae387cb66c7bbf79516a66b688c2c"}
Sep 05 02:18:32.609765 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.609683    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wxqrh" podStartSLOduration=6.716587099 podStartE2EDuration="21.609670533s" totalImagesPullingTime="14.893083434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:32.609183522 +0000 UTC m=+22.808546514" watchObservedRunningTime="2026-09-05 02:18:32.609670533 +0000 UTC m=+22.809033525"
Sep 05 02:18:32.609924 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.609905    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-8.ec2.internal" podStartSLOduration=21.60990047 podStartE2EDuration="21.60990047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:15.591354272 +0000 UTC m=+5.790717271" watchObservedRunningTime="2026-09-05 02:18:32.60990047 +0000 UTC m=+22.809263465"
Sep 05 02:18:32.626321 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.626272    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gc8cc" podStartSLOduration=4.384013701 podStartE2EDuration="22.626254989s" totalImagesPullingTime="18.242241288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:32.625746557 +0000 UTC m=+22.825109560" watchObservedRunningTime="2026-09-05 02:18:32.626254989 +0000 UTC m=+22.825617981"
Sep 05 02:18:32.667930 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.667866    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rsqww" podStartSLOduration=4.441702316 podStartE2EDuration="22.667843644s" totalImagesPullingTime="18.226141328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:32.667411496 +0000 UTC m=+22.866774490" watchObservedRunningTime="2026-09-05 02:18:32.667843644 +0000 UTC m=+22.867206639"
Sep 05 02:18:32.668112 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.668014    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f2sf6" podStartSLOduration=3.368826541 podStartE2EDuration="21.668008946s" totalImagesPullingTime="18.299182405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:32.642322072 +0000 UTC m=+22.841685089" watchObservedRunningTime="2026-09-05 02:18:32.668008946 +0000 UTC m=+22.867371942"
Sep 05 02:18:32.714190 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:32.714122    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-96gdl" podStartSLOduration=3.469906535 podStartE2EDuration="21.71407962s" totalImagesPullingTime="18.244173085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:32.71321842 +0000 UTC m=+22.912581413" watchObservedRunningTime="2026-09-05 02:18:32.71407962 +0000 UTC m=+22.913442615"
Sep 05 02:18:33.058879 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.058706    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:33.280149 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.280114    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:33.387073 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.386984    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:33.387073 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.387013    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:33.387599 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:33.387126    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:33.387599 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:33.387246    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:33.610314 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.610271    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h5dpn" event={"ID":"5c17b789-5a0b-4ebc-9dda-0ebca96bc357","Type":"ContainerStarted","Data":"ebbc62f565ecf37348aed94a610411f6fae525655722acd71aa5597cb1b9e80b"}
Sep 05 02:18:33.612427 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.612391    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc" event={"ID":"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89","Type":"ContainerStarted","Data":"9392ec7320bb62edcbc9362727196c01aaa6013858de95e1c4ad048d905c0c66"}
Sep 05 02:18:33.627584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:33.627513    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h5dpn" podStartSLOduration=4.390177027 podStartE2EDuration="22.627494561s" totalImagesPullingTime="18.237317534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:33.627136052 +0000 UTC m=+23.826499045" watchObservedRunningTime="2026-09-05 02:18:33.627494561 +0000 UTC m=+23.826857553"
Sep 05 02:18:34.235672 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:34.235475    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:33.280134332Z","UUID":"ca03a834-3c52-47d6-960b-46dbbeef3285","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:34.237445 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:34.237418    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:34.237604 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:34.237456    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:34.387125 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:34.387089    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:34.387619 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:34.387227    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:35.299866 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.299828    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:35.300537 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.300511    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:35.386960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.386929    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:35.386960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.386953    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:35.387158 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:35.387047    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:35.387425 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:35.387167    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:35.619723 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.619623    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc" event={"ID":"ad4dd761-a3cd-4bb3-8db9-62bc2388ad89","Type":"ContainerStarted","Data":"2c1a66c11d2310926961b4ee8f8cf900b46c52a76f939a1ad8c4198834e0a837"}
Sep 05 02:18:35.623445 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.623417    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"506d390f2ef3b7a059049a6d3eadb5504d4dd82f91e536dc02a0918fbf4165be"}
Sep 05 02:18:35.624159 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.624126    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wxqrh"
Sep 05 02:18:35.638985 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:35.638931    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpdlc" podStartSLOduration=5.071144771 podStartE2EDuration="25.638912027s" totalImagesPullingTime="20.567767256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:35.638428299 +0000 UTC m=+25.837791294" watchObservedRunningTime="2026-09-05 02:18:35.638912027 +0000 UTC m=+25.838275021"
Sep 05 02:18:36.386988 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:36.386956    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:36.387152 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:36.387083    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:37.387609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.387583    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:37.388124 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.387619    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:37.388124 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:37.387716    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:37.388124 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:37.387850    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:37.630588 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.630504    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" event={"ID":"687cd57d-c36b-4913-b647-39fcfe11fc70","Type":"ContainerStarted","Data":"554380f6eb6a15556735abf19fd9b9d47b45103ce9b42bbafddee1d2e2e882f5"}
Sep 05 02:18:37.630761 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.630737    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:37.632367 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.632346    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6862640-b4c3-47c5-a10b-13d7df1a9948" containerID="1344c0076049e8e01cffe3088ae9290a73f4c175c2daa0513a0385414dcfefc1" exitCode=0
Sep 05 02:18:37.632449 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.632414    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerDied","Data":"1344c0076049e8e01cffe3088ae9290a73f4c175c2daa0513a0385414dcfefc1"}
Sep 05 02:18:37.645720 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.645703    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:37.662130 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:37.662094    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-996cw" podStartSLOduration=8.356706784 podStartE2EDuration="26.662082203s" totalImagesPullingTime="18.305375419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:37.661941417 +0000 UTC m=+27.861304409" watchObservedRunningTime="2026-09-05 02:18:37.662082203 +0000 UTC m=+27.861445195"
Sep 05 02:18:38.388964 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:38.388929    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:38.389713 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:38.389055    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:38.635608 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:38.635345    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:38.635757 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:38.635674    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:38.649444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:38.649377    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:18:39.003263 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.003235    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2fzp6"]
Sep 05 02:18:39.003369 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.003339    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:39.003442 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:39.003422    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:39.009268 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.009243    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xmj5p"]
Sep 05 02:18:39.009365 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.009353    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:39.009482 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:39.009462    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:39.009788 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.009772    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8vlxt"]
Sep 05 02:18:39.009855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.009844    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:39.009932 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:39.009917    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:39.638382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.638353    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6862640-b4c3-47c5-a10b-13d7df1a9948" containerID="a51f75813f7de9da11e4ac0821711c72a24a8ddd831db10deee3c76725b92219" exitCode=0
Sep 05 02:18:39.638828 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:39.638413    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerDied","Data":"a51f75813f7de9da11e4ac0821711c72a24a8ddd831db10deee3c76725b92219"}
Sep 05 02:18:40.388257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:40.388228    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:40.388366 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:40.388337    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:40.388366 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:40.388344    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:40.388480 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:40.388372    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:40.388480 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:40.388431    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:40.388596 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:40.388509    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:40.642862 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:40.642824    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6862640-b4c3-47c5-a10b-13d7df1a9948" containerID="3090e7b22e27101160cee2d48f459dd010392450bb4c8c4fcef5fb3430ad2128" exitCode=0
Sep 05 02:18:40.643312 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:40.642899    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerDied","Data":"3090e7b22e27101160cee2d48f459dd010392450bb4c8c4fcef5fb3430ad2128"}
Sep 05 02:18:42.386918 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:42.386873    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:42.386918 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:42.386888    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:42.387476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:42.386990    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:42.387476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:42.387066    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:42.387476 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:42.387093    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:42.387476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:42.387190    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:44.023069 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.023036    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:44.023548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.023106    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:44.023548 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.023195    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:44.023548 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.023216    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:44.023548 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.023265    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret podName:a857aaf5-62f1-4114-a4e3-2b1ecad4ef84 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.023246826 +0000 UTC m=+66.222609806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret") pod "global-pull-secret-syncer-2fzp6" (UID: "a857aaf5-62f1-4114-a4e3-2b1ecad4ef84") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:44.023548 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.023284    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.023274523 +0000 UTC m=+66.222637495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:44.123389 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.123351    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:44.123574 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.123496    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:44.123574 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.123516    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:44.123574 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.123526    2553 projected.go:196] Error preparing data for projected volume kube-api-access-t66hs for pod openshift-network-diagnostics/network-check-target-8vlxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:44.123745 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.123594    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs podName:4a49dd59-7065-40d8-9e2a-51a12dd2b3d9 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.123576026 +0000 UTC m=+66.322938999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t66hs" (UniqueName: "kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs") pod "network-check-target-8vlxt" (UID: "4a49dd59-7065-40d8-9e2a-51a12dd2b3d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:44.387679 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.387647    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:44.387833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.387651    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:44.387833 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.387780    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fzp6" podUID="a857aaf5-62f1-4114-a4e3-2b1ecad4ef84"
Sep 05 02:18:44.387833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.387651    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:44.387969 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.387855    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xmj5p" podUID="7feca731-5171-453d-b2c6-9a5115d03b92"
Sep 05 02:18:44.387969 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.387899    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vlxt" podUID="4a49dd59-7065-40d8-9e2a-51a12dd2b3d9"
Sep 05 02:18:44.647631 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.647548    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-8.ec2.internal" event="NodeReady"
Sep 05 02:18:44.647762 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.647660    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:44.681666 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.681639    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-579dd5547b-2vc4k"]
Sep 05 02:18:44.710801 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.710778    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pcsnn"]
Sep 05 02:18:44.710936 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.710910    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.713540 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.713284    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:18:44.713776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.713755    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hnv4q\""
Sep 05 02:18:44.713869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.713792    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:18:44.713869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.713804    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:18:44.720124 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.720102    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:18:44.724854 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.724833    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579dd5547b-2vc4k"]
Sep 05 02:18:44.724948 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.724868    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gp8cp"]
Sep 05 02:18:44.724948 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.724933    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.727397 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.727379    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9hcm\""
Sep 05 02:18:44.727397 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.727382    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:44.727679 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.727666    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:44.747433 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.747407    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcsnn"]
Sep 05 02:18:44.747510 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.747461    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gp8cp"]
Sep 05 02:18:44.747510 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.747411    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.749933 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.749914    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:44.750028 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.749957    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:44.750235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.750220    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:44.750284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.750246    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl8b6\""
Sep 05 02:18:44.829305 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829285    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0ee1e15-56a4-4c32-9984-6545d50257ac-ca-trust-extracted\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829361 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829320    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kprx\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-kube-api-access-5kprx\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829361 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829346    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-config-volume\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.829429 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829411    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9j48\" (UniqueName: \"kubernetes.io/projected/9b4cb734-f2b5-4a9e-b322-d91f90d4e05a-kube-api-access-s9j48\") pod \"ingress-canary-gp8cp\" (UID: \"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a\") " pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.829466 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829452    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-certificates\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829497 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829486    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-image-registry-private-configuration\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829526 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829568 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829524    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-trusted-ca\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829568 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829546    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.829633 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829583    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l5x\" (UniqueName: \"kubernetes.io/projected/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-kube-api-access-52l5x\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.829665 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829630    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-installation-pull-secrets\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829665 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829659    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-tmp-dir\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.829727 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-bound-sa-token\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.829727 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.829697    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b4cb734-f2b5-4a9e-b322-d91f90d4e05a-cert\") pod \"ingress-canary-gp8cp\" (UID: \"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a\") " pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.930279 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930256    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0ee1e15-56a4-4c32-9984-6545d50257ac-ca-trust-extracted\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930288    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kprx\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-kube-api-access-5kprx\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930306    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-config-volume\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.930377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930338    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9j48\" (UniqueName: \"kubernetes.io/projected/9b4cb734-f2b5-4a9e-b322-d91f90d4e05a-kube-api-access-s9j48\") pod \"ingress-canary-gp8cp\" (UID: \"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a\") " pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.930519 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930376    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-certificates\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930519 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930416    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-image-registry-private-configuration\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930519 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930443    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930519 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930467    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-trusted-ca\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930519 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930489    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.930519 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930514    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l5x\" (UniqueName: \"kubernetes.io/projected/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-kube-api-access-52l5x\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930551    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-installation-pull-secrets\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930600    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-tmp-dir\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930637    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-bound-sa-token\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930659    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b4cb734-f2b5-4a9e-b322-d91f90d4e05a-cert\") pod \"ingress-canary-gp8cp\" (UID: \"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a\") " pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.930659    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.930721    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930736    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0ee1e15-56a4-4c32-9984-6545d50257ac-ca-trust-extracted\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.930830 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.930767    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.430751875 +0000 UTC m=+35.630114857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:18:44.931250 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.930975    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-config-volume\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.931250 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.931040    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:44.931250 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:44.931086    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.431069241 +0000 UTC m=+35.630432226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:44.931250 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.931231    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-certificates\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.931478 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.931317    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-tmp-dir\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.932095 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.932042    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-trusted-ca\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.934993 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.934971    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-image-registry-private-configuration\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.935080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.934988    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-installation-pull-secrets\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.935080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.935051    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b4cb734-f2b5-4a9e-b322-d91f90d4e05a-cert\") pod \"ingress-canary-gp8cp\" (UID: \"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a\") " pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.942851 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.942812    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9j48\" (UniqueName: \"kubernetes.io/projected/9b4cb734-f2b5-4a9e-b322-d91f90d4e05a-kube-api-access-s9j48\") pod \"ingress-canary-gp8cp\" (UID: \"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a\") " pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:44.942948 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.942899    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-bound-sa-token\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:44.942948 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.942927    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l5x\" (UniqueName: \"kubernetes.io/projected/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-kube-api-access-52l5x\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:44.943061 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:44.943016    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kprx\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-kube-api-access-5kprx\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:45.055484 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.055458    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gp8cp"
Sep 05 02:18:45.364447 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.364422    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gp8cp"]
Sep 05 02:18:45.367032 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:18:45.367005    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4cb734_f2b5_4a9e_b322_d91f90d4e05a.slice/crio-d3ed71cd42dd35a4deff183e22a23f912b4961eb0eeb657ab840a6a1daad4c09 WatchSource:0}: Error finding container d3ed71cd42dd35a4deff183e22a23f912b4961eb0eeb657ab840a6a1daad4c09: Status 404 returned error can't find the container with id d3ed71cd42dd35a4deff183e22a23f912b4961eb0eeb657ab840a6a1daad4c09
Sep 05 02:18:45.435422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.435386    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:45.435422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.435420    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:45.435614 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:45.435520    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:45.435614 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:45.435529    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:45.435614 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:45.435548    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:18:45.435614 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:45.435587    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:46.435555412 +0000 UTC m=+36.634918387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:45.435769 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:45.435625    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:18:46.435609987 +0000 UTC m=+36.634972968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:18:45.656169 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.656101    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gp8cp" event={"ID":"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a","Type":"ContainerStarted","Data":"d3ed71cd42dd35a4deff183e22a23f912b4961eb0eeb657ab840a6a1daad4c09"}
Sep 05 02:18:45.658959 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.658935    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6862640-b4c3-47c5-a10b-13d7df1a9948" containerID="e26bc802759d9bfd26fae2848367c53ead560bdf6fb492704e85090506c40e43" exitCode=0
Sep 05 02:18:45.659065 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:45.658977    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerDied","Data":"e26bc802759d9bfd26fae2848367c53ead560bdf6fb492704e85090506c40e43"}
Sep 05 02:18:46.387081 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.387044    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:18:46.387081 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.387066    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:18:46.387848 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.387058    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:18:46.391206 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.391180    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqlb9\""
Sep 05 02:18:46.391296 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.391208    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:46.391296 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.391220    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:46.391296 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.391181    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:46.391477 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.391464    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rhfc\""
Sep 05 02:18:46.391660 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.391642    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:46.443102 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.443077    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:46.443201 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.443109    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:46.443274 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:46.443225    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:46.443274 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:46.443235    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:46.443274 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:46.443253    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:18:46.443391 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:46.443279    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:48.443263948 +0000 UTC m=+38.642626932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:46.443391 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:46.443303    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:18:48.443286165 +0000 UTC m=+38.642649148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:18:46.665683 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.665596    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6862640-b4c3-47c5-a10b-13d7df1a9948" containerID="cd382de199391c1e55ff7e69b2e312d513578d13e2dd0a8164db539e709e3645" exitCode=0
Sep 05 02:18:46.665683 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:46.665670    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerDied","Data":"cd382de199391c1e55ff7e69b2e312d513578d13e2dd0a8164db539e709e3645"}
Sep 05 02:18:47.674091 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:47.674058    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zmrls" event={"ID":"d6862640-b4c3-47c5-a10b-13d7df1a9948","Type":"ContainerStarted","Data":"a65d9fbdba9b8f37f138b5d6b813fcf7bdf2dd2bbeb396a3c976863cc3f8007e"}
Sep 05 02:18:47.697145 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:47.697095    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zmrls" podStartSLOduration=8.864096539 podStartE2EDuration="37.697080418s" totalImagesPullingTime="28.832983879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:47.695295044 +0000 UTC m=+37.894658036" watchObservedRunningTime="2026-09-05 02:18:47.697080418 +0000 UTC m=+37.896443410"
Sep 05 02:18:48.460218 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:48.460185    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:48.460218 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:48.460218    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:48.460406 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:48.460306    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:48.460406 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:48.460321    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:48.460406 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:48.460338    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:18:48.460406 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:48.460356    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.460344024 +0000 UTC m=+42.659706994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:48.460406 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:48.460379    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.460366562 +0000 UTC m=+42.659729533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:18:48.677939 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:48.677759    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gp8cp" event={"ID":"9b4cb734-f2b5-4a9e-b322-d91f90d4e05a","Type":"ContainerStarted","Data":"5fda0814b5717686ed912460e0949debc6b9322c877bbadbe13aae2b34b55081"}
Sep 05 02:18:48.695257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:48.695210    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gp8cp" podStartSLOduration=2.225531425 podStartE2EDuration="4.695194136s" totalImagesPullingTime="2.469662711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:48.693946641 +0000 UTC m=+38.893309648" watchObservedRunningTime="2026-09-05 02:18:48.695194136 +0000 UTC m=+38.894557128"
Sep 05 02:18:52.487721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:52.487685    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:18:52.487721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:18:52.487721    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:18:52.488235 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:52.487864    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:52.488235 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:52.487887    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:18:52.488235 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:52.487889    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:52.488235 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:52.487945    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.487929494 +0000 UTC m=+50.687292470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:52.488235 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:18:52.487960    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.487953667 +0000 UTC m=+50.687316637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:19:00.546823 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:00.546788    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:19:00.546823 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:00.546824    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:19:00.547272 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:00.546935    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:00.547272 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:00.546942    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:00.547272 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:00.546960    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:19:00.547272 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:00.546991    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.546979128 +0000 UTC m=+66.746342099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:00.547272 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:00.547016    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.547003235 +0000 UTC m=+66.746366205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:19:10.653334 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:10.653304    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-996cw"
Sep 05 02:19:16.056875 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.056834    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:19:16.057311 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.056930    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:19:16.062362 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.059829    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:19:16.062362 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.059905    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:19:16.067108 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.067089    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:19:16.067192 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.067163    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs podName:7feca731-5171-453d-b2c6-9a5115d03b92 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:20.067145195 +0000 UTC m=+130.266508166 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs") pod "network-metrics-daemon-xmj5p" (UID: "7feca731-5171-453d-b2c6-9a5115d03b92") : secret "metrics-daemon-secret" not found
Sep 05 02:19:16.070831 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.070806    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a857aaf5-62f1-4114-a4e3-2b1ecad4ef84-original-pull-secret\") pod \"global-pull-secret-syncer-2fzp6\" (UID: \"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84\") " pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:19:16.111152 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.111127    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fzp6"
Sep 05 02:19:16.158239 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.158214    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:19:16.161147 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.161128    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:19:16.171624 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.171594    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:19:16.181456 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.181433    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66hs\" (UniqueName: \"kubernetes.io/projected/4a49dd59-7065-40d8-9e2a-51a12dd2b3d9-kube-api-access-t66hs\") pod \"network-check-target-8vlxt\" (UID: \"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9\") " pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:19:16.408695 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.408617    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rhfc\""
Sep 05 02:19:16.411212 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.411187    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2fzp6"]
Sep 05 02:19:16.416354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.416333    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:19:16.419065 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:19:16.419036    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda857aaf5_62f1_4114_a4e3_2b1ecad4ef84.slice/crio-dd3b0d9e2b2b343c9c1ab671edfd9c3e7841032a48e49e1b51bb03f774b72b9e WatchSource:0}: Error finding container dd3b0d9e2b2b343c9c1ab671edfd9c3e7841032a48e49e1b51bb03f774b72b9e: Status 404 returned error can't find the container with id dd3b0d9e2b2b343c9c1ab671edfd9c3e7841032a48e49e1b51bb03f774b72b9e
Sep 05 02:19:16.561254 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.561221    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:19:16.561254 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.561254    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:19:16.561498 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.561382    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:16.561498 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.561409    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:16.561498 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.561431    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:19:16.561498 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.561446    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:19:48.561429032 +0000 UTC m=+98.760792022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:16.561498 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:16.561482    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:19:48.56146505 +0000 UTC m=+98.760828042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:19:16.683892 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.683822    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8vlxt"]
Sep 05 02:19:16.686628 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:19:16.686604    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a49dd59_7065_40d8_9e2a_51a12dd2b3d9.slice/crio-943d8988117f7b7add71812cc5442e6e0148317797ecf8f159be9902249354d5 WatchSource:0}: Error finding container 943d8988117f7b7add71812cc5442e6e0148317797ecf8f159be9902249354d5: Status 404 returned error can't find the container with id 943d8988117f7b7add71812cc5442e6e0148317797ecf8f159be9902249354d5
Sep 05 02:19:16.741833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.741804    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8vlxt" event={"ID":"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9","Type":"ContainerStarted","Data":"943d8988117f7b7add71812cc5442e6e0148317797ecf8f159be9902249354d5"}
Sep 05 02:19:16.743117 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:16.743090    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2fzp6" event={"ID":"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84","Type":"ContainerStarted","Data":"dd3b0d9e2b2b343c9c1ab671edfd9c3e7841032a48e49e1b51bb03f774b72b9e"}
Sep 05 02:19:21.757221 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:21.757182    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8vlxt" event={"ID":"4a49dd59-7065-40d8-9e2a-51a12dd2b3d9","Type":"ContainerStarted","Data":"2b8cf092a47e9b36b6e326fbcb13f634d79df8ea2efe69ef62a98853adba02da"}
Sep 05 02:19:21.757688 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:21.757308    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:19:21.758708 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:21.758686    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2fzp6" event={"ID":"a857aaf5-62f1-4114-a4e3-2b1ecad4ef84","Type":"ContainerStarted","Data":"8eab4c8e8f8e11490120b9eaad83c17f9e17d3b260dbf869e6f52e83a26d653c"}
Sep 05 02:19:21.783232 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:21.783189    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8vlxt" podStartSLOduration=66.492811865 podStartE2EDuration="1m10.78317669s" totalImagesPullingTime="4.290364825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:21.783018916 +0000 UTC m=+71.982381911" watchObservedRunningTime="2026-09-05 02:19:21.78317669 +0000 UTC m=+71.982539683"
Sep 05 02:19:21.802030 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:21.801996    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2fzp6" podStartSLOduration=66.245503821 podStartE2EDuration="1m10.801984139s" totalImagesPullingTime="4.556480318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:21.801919728 +0000 UTC m=+72.001282723" watchObservedRunningTime="2026-09-05 02:19:21.801984139 +0000 UTC m=+72.001347132"
Sep 05 02:19:48.575059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:48.575028    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:19:48.575059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:48.575060    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:19:48.575378 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:48.575143    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:48.575378 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:48.575179    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:48.575378 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:48.575198    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579dd5547b-2vc4k: secret "image-registry-tls" not found
Sep 05 02:19:48.575378 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:48.575202    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls podName:b72ef229-be3c-4e6e-8d1f-59488d8a46ff nodeName:}" failed. No retries permitted until 2026-09-05 02:20:52.575189882 +0000 UTC m=+162.774552853 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls") pod "dns-default-pcsnn" (UID: "b72ef229-be3c-4e6e-8d1f-59488d8a46ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:48.575378 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:48.575251    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls podName:a0ee1e15-56a4-4c32-9984-6545d50257ac nodeName:}" failed. No retries permitted until 2026-09-05 02:20:52.575237428 +0000 UTC m=+162.774600399 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls") pod "image-registry-579dd5547b-2vc4k" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac") : secret "image-registry-tls" not found
Sep 05 02:19:52.763824 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:52.763795    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8vlxt"
Sep 05 02:19:56.517935 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.517900    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7978cdc4b-rn5tr"]
Sep 05 02:19:56.520523 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.520503    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.528166 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.528148    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 05 02:19:56.528446 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.528431    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5vjrb\""
Sep 05 02:19:56.528992 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.528978    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.529206 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.529192    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 05 02:19:56.529423 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.529410    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 05 02:19:56.529895 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.529883    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.529945 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.529894    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 05 02:19:56.548309 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.548283    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7978cdc4b-rn5tr"]
Sep 05 02:19:56.626001 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.625977    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-stats-auth\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.626161 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.626135    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.626257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.626176    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7nz\" (UniqueName: \"kubernetes.io/projected/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-kube-api-access-dj7nz\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.626257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.626196    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-default-certificate\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.626257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.626212    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.727482 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.727451    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-stats-auth\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.727636 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.727493    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.727636 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.727515    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7nz\" (UniqueName: \"kubernetes.io/projected/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-kube-api-access-dj7nz\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.727636 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.727544    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-default-certificate\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.727636 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.727582    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.727815 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:56.727633    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:19:56.727815 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:56.727704    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:57.227689532 +0000 UTC m=+107.427052503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : secret "router-metrics-certs-default" not found
Sep 05 02:19:56.727815 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:56.727747    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:57.227732499 +0000 UTC m=+107.427095474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : configmap references non-existent config key: service-ca.crt
Sep 05 02:19:56.729917 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.729896    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-default-certificate\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.730085 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.730065    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-stats-auth\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:56.746602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:56.746575    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7nz\" (UniqueName: \"kubernetes.io/projected/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-kube-api-access-dj7nz\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:57.231138 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:57.231098    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:57.231138 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:57.231141    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:57.231344 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:57.231242    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:19:57.231344 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:57.231257    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.231244741 +0000 UTC m=+108.430607712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : configmap references non-existent config key: service-ca.crt
Sep 05 02:19:57.231344 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:57.231304    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.231288843 +0000 UTC m=+108.430651817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : secret "router-metrics-certs-default" not found
Sep 05 02:19:58.237991 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:58.237959    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:58.237991 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:19:58.237994    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:19:58.238394 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:58.238105    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:00.238089868 +0000 UTC m=+110.437452839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : configmap references non-existent config key: service-ca.crt
Sep 05 02:19:58.238394 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:58.238118    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:19:58.238394 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:19:58.238166    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:00.238154748 +0000 UTC m=+110.437517720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : secret "router-metrics-certs-default" not found
Sep 05 02:20:00.249254 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:00.249222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:00.249758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:00.249267    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:00.249758 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:00.249373    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:20:00.249758 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:00.249402    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.249381012 +0000 UTC m=+114.448743989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : configmap references non-existent config key: service-ca.crt
Sep 05 02:20:00.249758 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:00.249436    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.249422716 +0000 UTC m=+114.448785691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : secret "router-metrics-certs-default" not found
Sep 05 02:20:01.196153 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.196124    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jblb8"]
Sep 05 02:20:01.198081 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.198065    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.201399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.201372    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:20:01.201532 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.201372    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:20:01.202110 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.202093    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:20:01.202664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.202650    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:20:01.202754 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.202720    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mx6qs\""
Sep 05 02:20:01.212016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.211993    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jblb8"]
Sep 05 02:20:01.256485 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.256466    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.256794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.256512    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-data-volume\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.256794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.256531    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-crio-socket\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.256794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.256632    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqwg\" (UniqueName: \"kubernetes.io/projected/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-api-access-ssqwg\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.256794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.256681    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357298 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357263    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-data-volume\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357298 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-crio-socket\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357492 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357323    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqwg\" (UniqueName: \"kubernetes.io/projected/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-api-access-ssqwg\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357492 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357403    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-crio-socket\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357491    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357644 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357595    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357644 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.357610    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.357719 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.357661    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:01.857647832 +0000 UTC m=+112.057010803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.357719 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.357658    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-data-volume\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.357719 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.357713    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.357835 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.357766    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:01.857751458 +0000 UTC m=+112.057114431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.366164 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.366143    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqwg\" (UniqueName: \"kubernetes.io/projected/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-api-access-ssqwg\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.861551 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.861518    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.861731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:01.861581    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:01.861731 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.861643    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.861731 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.861686    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.861731 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.861732    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:02.861719427 +0000 UTC m=+113.061082398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.861872 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:01.861744    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:02.86173865 +0000 UTC m=+113.061101621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.869012 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:02.868982    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:02.869373 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:02.869028    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:02.869373 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:02.869090    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.869373 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:02.869146    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.869131058 +0000 UTC m=+115.068494034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.869373 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:02.869172    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:02.869373 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:02.869216    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.86919989 +0000 UTC m=+115.068562862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:03.950787 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:03.950758    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-96gdl_0d380942-90b8-482d-a917-d4d3435fb445/dns-node-resolver/0.log"
Sep 05 02:20:04.280660 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:04.280630    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:04.280814 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:04.280677    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:04.280814 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.280785    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:12.280767896 +0000 UTC m=+122.480130867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : configmap references non-existent config key: service-ca.crt
Sep 05 02:20:04.280814 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.280785    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:20:04.280943 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.280850    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs podName:ae4b62c8-6434-4a5e-a9e2-b54c10be90b9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:12.280836659 +0000 UTC m=+122.480199636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs") pod "router-default-7978cdc4b-rn5tr" (UID: "ae4b62c8-6434-4a5e-a9e2-b54c10be90b9") : secret "router-metrics-certs-default" not found
Sep 05 02:20:04.885246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:04.885211    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:04.885408 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:04.885315    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:04.885408 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.885364    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:04.885408 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.885387    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:04.885529 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.885437    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:08.885419111 +0000 UTC m=+119.084782085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:04.885529 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:04.885451    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:08.885445375 +0000 UTC m=+119.084808346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:05.148531 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:05.148467    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rsqww_a2a60161-98fa-4b9d-87a1-457e09a1b7fa/node-ca/0.log"
Sep 05 02:20:05.746373 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:05.746347    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gp8cp_9b4cb734-f2b5-4a9e-b322-d91f90d4e05a/serve-healthcheck-canary/0.log"
Sep 05 02:20:08.915036 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:08.915000    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:08.915476 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:08.915086    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:08.915476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:08.915143    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:08.915476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:08.915203    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:16.915188839 +0000 UTC m=+127.114551810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:08.915476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:08.915150    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:08.915476 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:08.915269    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm podName:d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:16.915257952 +0000 UTC m=+127.114620923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jblb8" (UID: "d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:12.339522 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.339488    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:12.339894 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.339587    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:12.340076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.340055    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-service-ca-bundle\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:12.341724 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.341695    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae4b62c8-6434-4a5e-a9e2-b54c10be90b9-metrics-certs\") pod \"router-default-7978cdc4b-rn5tr\" (UID: \"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9\") " pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:12.431365 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.431338    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5vjrb\""
Sep 05 02:20:12.439490 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.439471    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:12.716116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.716045    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7978cdc4b-rn5tr"]
Sep 05 02:20:12.719132 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:12.719103    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4b62c8_6434_4a5e_a9e2_b54c10be90b9.slice/crio-f01389550baaf1d993e9aa4ff66108b387ec536581d6d568fee884eebbce5a99 WatchSource:0}: Error finding container f01389550baaf1d993e9aa4ff66108b387ec536581d6d568fee884eebbce5a99: Status 404 returned error can't find the container with id f01389550baaf1d993e9aa4ff66108b387ec536581d6d568fee884eebbce5a99
Sep 05 02:20:12.877416 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.877382    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7978cdc4b-rn5tr" event={"ID":"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9","Type":"ContainerStarted","Data":"3dbdf14c24e87781aa1b3f847283960a24dfbd6950359c2525a6c89ce83b8519"}
Sep 05 02:20:12.877589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.877429    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7978cdc4b-rn5tr" event={"ID":"ae4b62c8-6434-4a5e-a9e2-b54c10be90b9","Type":"ContainerStarted","Data":"f01389550baaf1d993e9aa4ff66108b387ec536581d6d568fee884eebbce5a99"}
Sep 05 02:20:12.902708 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:12.902662    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7978cdc4b-rn5tr" podStartSLOduration=16.902647776 podStartE2EDuration="16.902647776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:12.901496107 +0000 UTC m=+123.100859094" watchObservedRunningTime="2026-09-05 02:20:12.902647776 +0000 UTC m=+123.102010769"
Sep 05 02:20:13.440510 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:13.440475    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:13.442646 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:13.442623    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:13.879784 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:13.879753    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:13.880885 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:13.880859    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7978cdc4b-rn5tr"
Sep 05 02:20:16.976739 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:16.976680    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:16.976739 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:16.976748    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:16.977230 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:16.977208    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:16.979052 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:16.979034    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jblb8\" (UID: \"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5\") " pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:17.109996 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:17.109966    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mx6qs\""
Sep 05 02:20:17.117888 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:17.117868    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jblb8"
Sep 05 02:20:17.390670 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:17.390639    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jblb8"]
Sep 05 02:20:17.393627 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:17.393598    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86d4179_8e90_4785_8b3d_0c4ecb9cf1b5.slice/crio-b90f3147ed9be64fed68524e60c6b9833862c8cca4c37a8b4477bff88916e068 WatchSource:0}: Error finding container b90f3147ed9be64fed68524e60c6b9833862c8cca4c37a8b4477bff88916e068: Status 404 returned error can't find the container with id b90f3147ed9be64fed68524e60c6b9833862c8cca4c37a8b4477bff88916e068
Sep 05 02:20:17.890752 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:17.890711    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jblb8" event={"ID":"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5","Type":"ContainerStarted","Data":"750a6f1ce6ce41a252baa6b1a3ec15933b94400854ef9e4716b3c04548859a7e"}
Sep 05 02:20:17.890910 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:17.890773    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jblb8" event={"ID":"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5","Type":"ContainerStarted","Data":"b90f3147ed9be64fed68524e60c6b9833862c8cca4c37a8b4477bff88916e068"}
Sep 05 02:20:18.896763 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:18.896722    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jblb8" event={"ID":"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5","Type":"ContainerStarted","Data":"51a6a183fe5ae5ba1f38b7157fa1729e407790b4157972c990c62097a98ef301"}
Sep 05 02:20:19.902197 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:19.902160    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jblb8" event={"ID":"d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5","Type":"ContainerStarted","Data":"3569171bbc9bb50504dd33cf12deff5a3cde15918b3ed46494e7694de916fb3b"}
Sep 05 02:20:19.922269 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:19.922228    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jblb8" podStartSLOduration=16.976926381 podStartE2EDuration="18.922215468s" totalImagesPullingTime="1.945289087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:20:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:19.921666985 +0000 UTC m=+130.121029977" watchObservedRunningTime="2026-09-05 02:20:19.922215468 +0000 UTC m=+130.121578460"
Sep 05 02:20:20.103770 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:20.103729    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:20:20.105935 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:20.105909    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7feca731-5171-453d-b2c6-9a5115d03b92-metrics-certs\") pod \"network-metrics-daemon-xmj5p\" (UID: \"7feca731-5171-453d-b2c6-9a5115d03b92\") " pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:20:20.301131 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:20.301105    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqlb9\""
Sep 05 02:20:20.308342 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:20.308322    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xmj5p"
Sep 05 02:20:20.574898 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:20.574832    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xmj5p"]
Sep 05 02:20:20.577709 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:20.577680    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7feca731_5171_453d_b2c6_9a5115d03b92.slice/crio-217e0231240e8ddb50d60663b9099de7b389d90b614a983b49895d9cb752f80d WatchSource:0}: Error finding container 217e0231240e8ddb50d60663b9099de7b389d90b614a983b49895d9cb752f80d: Status 404 returned error can't find the container with id 217e0231240e8ddb50d60663b9099de7b389d90b614a983b49895d9cb752f80d
Sep 05 02:20:20.906603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:20.906519    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmj5p" event={"ID":"7feca731-5171-453d-b2c6-9a5115d03b92","Type":"ContainerStarted","Data":"217e0231240e8ddb50d60663b9099de7b389d90b614a983b49895d9cb752f80d"}
Sep 05 02:20:21.911216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:21.911131    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmj5p" event={"ID":"7feca731-5171-453d-b2c6-9a5115d03b92","Type":"ContainerStarted","Data":"9c75668bed07b2bcba305630d3d923a801ec52d1da377615dfb7df13aaa98eb7"}
Sep 05 02:20:21.911216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:21.911188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xmj5p" event={"ID":"7feca731-5171-453d-b2c6-9a5115d03b92","Type":"ContainerStarted","Data":"4fc39ee0a0dd552864e6d7805757db0d68dfccf1c506e0d576b9f38329d785cc"}
Sep 05 02:20:21.933122 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:21.933074    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xmj5p" podStartSLOduration=129.972955457 podStartE2EDuration="2m10.933059026s" totalImagesPullingTime="960.103569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:21.932725306 +0000 UTC m=+132.132088299" watchObservedRunningTime="2026-09-05 02:20:21.933059026 +0000 UTC m=+132.132422018"
Sep 05 02:20:24.358941 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.358909    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qvr68"]
Sep 05 02:20:24.361918 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.361896    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.365995 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.365964    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 05 02:20:24.366092 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.366021    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q5w5x\""
Sep 05 02:20:24.366092 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.366041    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 05 02:20:24.376433 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.376411    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qvr68"]
Sep 05 02:20:24.436891 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.436863    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6dd08154-87bf-435e-b2cc-827a2419b85f-nginx-conf\") pod \"networking-console-plugin-79977f9844-qvr68\" (UID: \"6dd08154-87bf-435e-b2cc-827a2419b85f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.437007 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.436925    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dd08154-87bf-435e-b2cc-827a2419b85f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qvr68\" (UID: \"6dd08154-87bf-435e-b2cc-827a2419b85f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.537338 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.537305    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6dd08154-87bf-435e-b2cc-827a2419b85f-nginx-conf\") pod \"networking-console-plugin-79977f9844-qvr68\" (UID: \"6dd08154-87bf-435e-b2cc-827a2419b85f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.537466 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.537377    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dd08154-87bf-435e-b2cc-827a2419b85f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qvr68\" (UID: \"6dd08154-87bf-435e-b2cc-827a2419b85f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.537976 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.537945    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6dd08154-87bf-435e-b2cc-827a2419b85f-nginx-conf\") pod \"networking-console-plugin-79977f9844-qvr68\" (UID: \"6dd08154-87bf-435e-b2cc-827a2419b85f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.539621 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.539602    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dd08154-87bf-435e-b2cc-827a2419b85f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qvr68\" (UID: \"6dd08154-87bf-435e-b2cc-827a2419b85f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.670416 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.670351    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68"
Sep 05 02:20:24.938166 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:24.938120    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qvr68"]
Sep 05 02:20:24.940588 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:24.940546    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd08154_87bf_435e_b2cc_827a2419b85f.slice/crio-90b4f5ea215cdb8687f9bc50b6121777e07ff753a4d1ba1b66c9eea44b8634fa WatchSource:0}: Error finding container 90b4f5ea215cdb8687f9bc50b6121777e07ff753a4d1ba1b66c9eea44b8634fa: Status 404 returned error can't find the container with id 90b4f5ea215cdb8687f9bc50b6121777e07ff753a4d1ba1b66c9eea44b8634fa
Sep 05 02:20:25.924386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:25.924343    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68" event={"ID":"6dd08154-87bf-435e-b2cc-827a2419b85f","Type":"ContainerStarted","Data":"90b4f5ea215cdb8687f9bc50b6121777e07ff753a4d1ba1b66c9eea44b8634fa"}
Sep 05 02:20:26.928409 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:26.928371    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68" event={"ID":"6dd08154-87bf-435e-b2cc-827a2419b85f","Type":"ContainerStarted","Data":"3485739f0aa9716c8bb16f4ee78d964e79dc1058441e4411bc8a7a4f7a2f4a51"}
Sep 05 02:20:26.945032 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:26.944976    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-qvr68" podStartSLOduration=1.710877025 podStartE2EDuration="2.944956745s" totalImagesPullingTime="1.23407972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:26.943451114 +0000 UTC m=+137.142814118" watchObservedRunningTime="2026-09-05 02:20:26.944956745 +0000 UTC m=+137.144319737"
Sep 05 02:20:31.015064 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.015038    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"]
Sep 05 02:20:31.018965 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.018949    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:31.021613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.021586    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-56jfg\""
Sep 05 02:20:31.021723 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.021642    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 05 02:20:31.028827 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.028803    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"]
Sep 05 02:20:31.086284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.086255    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea0af986-d6ca-4e86-b651-180096161852-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zk6kc\" (UID: \"ea0af986-d6ca-4e86-b651-180096161852\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:31.186639 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.186612    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea0af986-d6ca-4e86-b651-180096161852-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zk6kc\" (UID: \"ea0af986-d6ca-4e86-b651-180096161852\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:31.188865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.188847    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea0af986-d6ca-4e86-b651-180096161852-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zk6kc\" (UID: \"ea0af986-d6ca-4e86-b651-180096161852\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:31.327498 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.327463    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:31.598354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.598286    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"]
Sep 05 02:20:31.602161 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:31.602138    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0af986_d6ca_4e86_b651_180096161852.slice/crio-e65a0ffedef6cb96fa25d275dc662662391b0e466c45998835a1b9dba0eafcd7 WatchSource:0}: Error finding container e65a0ffedef6cb96fa25d275dc662662391b0e466c45998835a1b9dba0eafcd7: Status 404 returned error can't find the container with id e65a0ffedef6cb96fa25d275dc662662391b0e466c45998835a1b9dba0eafcd7
Sep 05 02:20:31.943823 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:31.943745    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc" event={"ID":"ea0af986-d6ca-4e86-b651-180096161852","Type":"ContainerStarted","Data":"e65a0ffedef6cb96fa25d275dc662662391b0e466c45998835a1b9dba0eafcd7"}
Sep 05 02:20:32.948627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:32.948530    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc" event={"ID":"ea0af986-d6ca-4e86-b651-180096161852","Type":"ContainerStarted","Data":"c0b3db8a4f57455b170d647ccd44000e17c7e537664997b4c012493ac6611041"}
Sep 05 02:20:32.949019 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:32.948733    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:32.953112 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:32.953085    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc"
Sep 05 02:20:32.964135 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:32.964093    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zk6kc" podStartSLOduration=1.957095566 podStartE2EDuration="2.964081136s" totalImagesPullingTime="1.00698557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:32.963537263 +0000 UTC m=+143.162900280" watchObservedRunningTime="2026-09-05 02:20:32.964081136 +0000 UTC m=+143.163444128"
Sep 05 02:20:34.077895 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.077866    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-mq44t"]
Sep 05 02:20:34.080811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.080796    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.083449 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.083425    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wznsv\""
Sep 05 02:20:34.084761 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.084732    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 05 02:20:34.084869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.084817    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:20:34.084869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.084827    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:20:34.084869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.084835    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 05 02:20:34.085010 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.084917    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:20:34.090052 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.090033    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-mq44t"]
Sep 05 02:20:34.208625 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.208590    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.208625 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.208627    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.208813 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.208647    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.208813 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.208749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bbb\" (UniqueName: \"kubernetes.io/projected/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-kube-api-access-p6bbb\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.309327 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.309291    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.309327 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.309331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.309463 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.309349    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.309463 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.309409    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbb\" (UniqueName: \"kubernetes.io/projected/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-kube-api-access-p6bbb\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.310001 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.309981    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.311651 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.311631    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.311729 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.311702    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.317612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.317592    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bbb\" (UniqueName: \"kubernetes.io/projected/a5977287-5f4c-4ea2-b4e8-424f7f329c4c-kube-api-access-p6bbb\") pod \"prometheus-operator-6f4946db56-mq44t\" (UID: \"a5977287-5f4c-4ea2-b4e8-424f7f329c4c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.389152 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.389102    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t"
Sep 05 02:20:34.657251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.657146    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-mq44t"]
Sep 05 02:20:34.659532 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:34.659507    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5977287_5f4c_4ea2_b4e8_424f7f329c4c.slice/crio-b38675d42f2da5ba4ff42a3dfe4a64fc8541393f63058be5699f17387c0617a4 WatchSource:0}: Error finding container b38675d42f2da5ba4ff42a3dfe4a64fc8541393f63058be5699f17387c0617a4: Status 404 returned error can't find the container with id b38675d42f2da5ba4ff42a3dfe4a64fc8541393f63058be5699f17387c0617a4
Sep 05 02:20:34.955690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:34.955620    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t" event={"ID":"a5977287-5f4c-4ea2-b4e8-424f7f329c4c","Type":"ContainerStarted","Data":"b38675d42f2da5ba4ff42a3dfe4a64fc8541393f63058be5699f17387c0617a4"}
Sep 05 02:20:36.963570 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:36.963537    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t" event={"ID":"a5977287-5f4c-4ea2-b4e8-424f7f329c4c","Type":"ContainerStarted","Data":"6c5a5e7e22651ab1985f64ec47faa77d24c22b0fc3135cb4e29569a497c9ec6d"}
Sep 05 02:20:36.963903 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:36.963636    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t" event={"ID":"a5977287-5f4c-4ea2-b4e8-424f7f329c4c","Type":"ContainerStarted","Data":"ff00dda7d10d718c01b39dcd26b9683c44d805b0972d86a3820bf45d86930e41"}
Sep 05 02:20:36.981187 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:36.981139    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-mq44t" podStartSLOduration=1.150028022 podStartE2EDuration="2.98112478s" totalImagesPullingTime="1.831096758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:36.979822769 +0000 UTC m=+147.179185802" watchObservedRunningTime="2026-09-05 02:20:36.98112478 +0000 UTC m=+147.180487773"
Sep 05 02:20:38.416608 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.416578    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"]
Sep 05 02:20:38.419814 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.419797    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.422596 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.422577    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:20:38.422702 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.422577    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 05 02:20:38.422702 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.422612    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8xczv\""
Sep 05 02:20:38.426808 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.426787    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9tfbs"]
Sep 05 02:20:38.429641 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.429627    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"]
Sep 05 02:20:38.429749 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.429736    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.432243 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.432224    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:20:38.432527 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.432502    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.432840 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.432819    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:20:38.433026 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.432993    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:20:38.433323 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.433306    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ld2zj\""
Sep 05 02:20:38.433684 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.433667    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"]
Sep 05 02:20:38.434892 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.434875    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 05 02:20:38.435331 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.435314    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fw4rp\""
Sep 05 02:20:38.435664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.435649    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 05 02:20:38.435966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.435950    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:20:38.450644 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.450624    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"]
Sep 05 02:20:38.540016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.539985    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkck\" (UniqueName: \"kubernetes.io/projected/82310afe-e173-4719-a2df-0333fb883f8e-kube-api-access-fhkck\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.540016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540017    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540036    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-wtmp\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540053    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.540236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540072    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgq2\" (UniqueName: \"kubernetes.io/projected/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-api-access-qqgq2\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.540236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540100    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.540236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540153    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-sys\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540203    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540256    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh99d\" (UniqueName: \"kubernetes.io/projected/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-kube-api-access-wh99d\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540331    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-root\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540390    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-tls\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540420    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82310afe-e173-4719-a2df-0333fb883f8e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540448    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-metrics-client-ca\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540516 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540506    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-textfile\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.540832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540537    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.540832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540621    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.540832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.540647    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641379 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641352    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-textfile\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641385    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641410    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641451    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkck\" (UniqueName: \"kubernetes.io/projected/82310afe-e173-4719-a2df-0333fb883f8e-kube-api-access-fhkck\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.641528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641486    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641510    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-wtmp\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641573    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgq2\" (UniqueName: \"kubernetes.io/projected/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-api-access-qqgq2\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641602    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641630    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-sys\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641657    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641709    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-wtmp\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641729    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641781    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh99d\" (UniqueName: \"kubernetes.io/projected/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-kube-api-access-wh99d\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641807    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:38.641812    2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 05 02:20:38.641852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641841    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-root\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641864    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-tls\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:38.641882    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-tls podName:3266367a-da1d-42be-8f8f-b7d05cf1a9c4 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:39.141864148 +0000 UTC m=+149.341227136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-gv6c5" (UID: "3266367a-da1d-42be-8f8f-b7d05cf1a9c4") : secret "kube-state-metrics-tls" not found
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641923    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82310afe-e173-4719-a2df-0333fb883f8e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:38.641933    2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641959    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-metrics-client-ca\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:38.641989    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-tls podName:42e4d112-aa98-4f8e-931a-487fcc4fa5c9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:39.141975976 +0000 UTC m=+149.341338950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-tls") pod "node-exporter-9tfbs" (UID: "42e4d112-aa98-4f8e-931a-487fcc4fa5c9") : secret "node-exporter-tls" not found
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:38.642040    2553 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:38.642072    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-tls podName:82310afe-e173-4719-a2df-0333fb883f8e nodeName:}" failed. No retries permitted until 2026-09-05 02:20:39.142061149 +0000 UTC m=+149.341424135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-9d6fc" (UID: "82310afe-e173-4719-a2df-0333fb883f8e") : secret "openshift-state-metrics-tls" not found
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.641837    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-textfile\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642133    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-sys\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642185    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-root\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642210    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.642359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642216    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.643027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642692    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.643027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642785    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-metrics-client-ca\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.643027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642801    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.643027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.642890    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82310afe-e173-4719-a2df-0333fb883f8e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.644731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.644703    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:38.644731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.644722    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.644856 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.644830    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.653271 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.653244    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh99d\" (UniqueName: \"kubernetes.io/projected/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-kube-api-access-wh99d\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:38.653874 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.653851    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgq2\" (UniqueName: \"kubernetes.io/projected/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-api-access-qqgq2\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:38.654839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:38.654822    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkck\" (UniqueName: \"kubernetes.io/projected/82310afe-e173-4719-a2df-0333fb883f8e-kube-api-access-fhkck\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:39.145014 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.144973    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-tls\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:39.145206 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.145066    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:39.145206 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.145119    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:39.147354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.147325    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/82310afe-e173-4719-a2df-0333fb883f8e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9d6fc\" (UID: \"82310afe-e173-4719-a2df-0333fb883f8e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:39.147460 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.147367    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4d112-aa98-4f8e-931a-487fcc4fa5c9-node-exporter-tls\") pod \"node-exporter-9tfbs\" (UID: \"42e4d112-aa98-4f8e-931a-487fcc4fa5c9\") " pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:39.147496 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.147489    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3266367a-da1d-42be-8f8f-b7d05cf1a9c4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gv6c5\" (UID: \"3266367a-da1d-42be-8f8f-b7d05cf1a9c4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:39.328703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.328674    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"
Sep 05 02:20:39.339752 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.339730    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tfbs"
Sep 05 02:20:39.346788 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.346586    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"
Sep 05 02:20:39.353211 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:39.353180    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e4d112_aa98_4f8e_931a_487fcc4fa5c9.slice/crio-5f4c68bcbbb5ed869499e7897da0f6583bd43f7cb3b59c132b2f785a826a4c4c WatchSource:0}: Error finding container 5f4c68bcbbb5ed869499e7897da0f6583bd43f7cb3b59c132b2f785a826a4c4c: Status 404 returned error can't find the container with id 5f4c68bcbbb5ed869499e7897da0f6583bd43f7cb3b59c132b2f785a826a4c4c
Sep 05 02:20:39.487884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.487855    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:39.492776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.492757    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.495709 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.495200    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pwrbs\""
Sep 05 02:20:39.495709 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.495324    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 02:20:39.495709 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.495420    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 02:20:39.495950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.495733    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 02:20:39.495950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.495930    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 02:20:39.496317 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.496078    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 02:20:39.496317 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.496128    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 02:20:39.496317 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.496233    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 02:20:39.496530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.496506    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 02:20:39.501305 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.501286    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 02:20:39.505254 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.505234    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:39.610193 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.610005    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc"]
Sep 05 02:20:39.612069 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:39.612042    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82310afe_e173_4719_a2df_0333fb883f8e.slice/crio-5cc65ba05384f2158189aeff99aa21af73f4f0d8bd773878e20c3b70361b9e07 WatchSource:0}: Error finding container 5cc65ba05384f2158189aeff99aa21af73f4f0d8bd773878e20c3b70361b9e07: Status 404 returned error can't find the container with id 5cc65ba05384f2158189aeff99aa21af73f4f0d8bd773878e20c3b70361b9e07
Sep 05 02:20:39.631852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.631827    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5"]
Sep 05 02:20:39.636578 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:39.636535    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3266367a_da1d_42be_8f8f_b7d05cf1a9c4.slice/crio-2f2e55c534973231e04add7aff3f27124c3b5b70aca8e0f36a99ed0eac870662 WatchSource:0}: Error finding container 2f2e55c534973231e04add7aff3f27124c3b5b70aca8e0f36a99ed0eac870662: Status 404 returned error can't find the container with id 2f2e55c534973231e04add7aff3f27124c3b5b70aca8e0f36a99ed0eac870662
Sep 05 02:20:39.650354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650333    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650426 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650360    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g628x\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-kube-api-access-g628x\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650426 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650380    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650426 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650406    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-config-out\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650533 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650432    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650533 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650466    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650533 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650495    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650645 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650533    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650645 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650549    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650645 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650582    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650645 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650604    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650840 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650654    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-web-config\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.650840 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.650682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751215 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751191    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751330 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751244    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751330 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751330 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751304    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751330 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751326    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751385    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-web-config\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751426    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751459    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751486    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g628x\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-kube-api-access-g628x\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751515    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751799 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751584    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-config-out\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751799 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751613    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.751799 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.751645    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.752463 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.752282    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.753181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.753152    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.753443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.753420    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.753974 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.753950    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.755113 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.755070    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.755691 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.755672    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-web-config\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.755691 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.755683    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.756078 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.756032    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.756174 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.756148    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.756316 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.756253    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-config-out\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.756316 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.756308    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.756505 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.756486    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.759196 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.759175    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g628x\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-kube-api-access-g628x\") pod \"alertmanager-main-0\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.803467 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.803446    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:39.974023 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.973959    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tfbs" event={"ID":"42e4d112-aa98-4f8e-931a-487fcc4fa5c9","Type":"ContainerStarted","Data":"5f4c68bcbbb5ed869499e7897da0f6583bd43f7cb3b59c132b2f785a826a4c4c"}
Sep 05 02:20:39.975027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.975006    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5" event={"ID":"3266367a-da1d-42be-8f8f-b7d05cf1a9c4","Type":"ContainerStarted","Data":"2f2e55c534973231e04add7aff3f27124c3b5b70aca8e0f36a99ed0eac870662"}
Sep 05 02:20:39.976574 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.976543    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc" event={"ID":"82310afe-e173-4719-a2df-0333fb883f8e","Type":"ContainerStarted","Data":"f8dda7173e95b3c9706a56f992c9ffa93936729fc67b9ac9cf35f5b40592f8cd"}
Sep 05 02:20:39.976647 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.976600    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc" event={"ID":"82310afe-e173-4719-a2df-0333fb883f8e","Type":"ContainerStarted","Data":"6da8c1886e85c30b0870d3e29d41a109a29f1dfd7c6839abd34b345fef57b189"}
Sep 05 02:20:39.976647 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:39.976619    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc" event={"ID":"82310afe-e173-4719-a2df-0333fb883f8e","Type":"ContainerStarted","Data":"5cc65ba05384f2158189aeff99aa21af73f4f0d8bd773878e20c3b70361b9e07"}
Sep 05 02:20:40.084978 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:40.084956    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:40.086524 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:40.086504    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfda1ed_6c0a_4702_9992_7466df7421a5.slice/crio-a409b0191c21a26662fd246c58de7d2946fba30390624576a9e26c574d29db91 WatchSource:0}: Error finding container a409b0191c21a26662fd246c58de7d2946fba30390624576a9e26c574d29db91: Status 404 returned error can't find the container with id a409b0191c21a26662fd246c58de7d2946fba30390624576a9e26c574d29db91
Sep 05 02:20:40.984316 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:40.984282    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"a409b0191c21a26662fd246c58de7d2946fba30390624576a9e26c574d29db91"}
Sep 05 02:20:41.493682 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.493652    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"]
Sep 05 02:20:41.497628 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.497605    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.500339 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.500300    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pqqxg\""
Sep 05 02:20:41.500339 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.500323    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b3akfnknimb4g\""
Sep 05 02:20:41.500521 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.500371    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 05 02:20:41.500521 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.500307    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 05 02:20:41.500521 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.500300    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 05 02:20:41.501345 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.501290    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 05 02:20:41.501345 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.501306    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 05 02:20:41.509943 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.509486    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"]
Sep 05 02:20:41.667246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667222    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-tls\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667375 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667257    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667375 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667280    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-grpc-tls\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667375 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcv95\" (UniqueName: \"kubernetes.io/projected/510f85ce-e160-4340-b969-857c4c9a15e4-kube-api-access-jcv95\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667481 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667396    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667628 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667603    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/510f85ce-e160-4340-b969-857c4c9a15e4-metrics-client-ca\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667668 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.667700 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.667669    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.743171 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.743135    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-rvl98"]
Sep 05 02:20:41.746669 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.746646    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:20:41.749628 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.749588    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:20:41.749728 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.749611    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:20:41.750345 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.750329    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gkf2r\""
Sep 05 02:20:41.761424 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.761400    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rvl98"]
Sep 05 02:20:41.768350 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768325    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-grpc-tls\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcv95\" (UniqueName: \"kubernetes.io/projected/510f85ce-e160-4340-b969-857c4c9a15e4-kube-api-access-jcv95\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768423    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768465    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/510f85ce-e160-4340-b969-857c4c9a15e4-metrics-client-ca\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768494    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768536    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768780 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768616    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-tls\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.768780 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.768650    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.769910 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.769883    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/510f85ce-e160-4340-b969-857c4c9a15e4-metrics-client-ca\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.773163 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.773128    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.773482 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.773440    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.773677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.773653    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-tls\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.774204 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.774183    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.774809 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.774789    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-grpc-tls\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.775646 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.775626    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/510f85ce-e160-4340-b969-857c4c9a15e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.781785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.781765    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcv95\" (UniqueName: \"kubernetes.io/projected/510f85ce-e160-4340-b969-857c4c9a15e4-kube-api-access-jcv95\") pod \"thanos-querier-64bfc7bdcb-nqvfn\" (UID: \"510f85ce-e160-4340-b969-857c4c9a15e4\") " pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.809354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.809322    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:41.869223 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.869098    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdh46\" (UniqueName: \"kubernetes.io/projected/965dc0fc-5506-42dd-a079-0d548fdd31c4-kube-api-access-wdh46\") pod \"downloads-85d98c998f-rvl98\" (UID: \"965dc0fc-5506-42dd-a079-0d548fdd31c4\") " pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:20:41.970138 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.970104    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh46\" (UniqueName: \"kubernetes.io/projected/965dc0fc-5506-42dd-a079-0d548fdd31c4-kube-api-access-wdh46\") pod \"downloads-85d98c998f-rvl98\" (UID: \"965dc0fc-5506-42dd-a079-0d548fdd31c4\") " pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:20:41.980368 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.980340    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh46\" (UniqueName: \"kubernetes.io/projected/965dc0fc-5506-42dd-a079-0d548fdd31c4-kube-api-access-wdh46\") pod \"downloads-85d98c998f-rvl98\" (UID: \"965dc0fc-5506-42dd-a079-0d548fdd31c4\") " pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:20:41.989778 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.989754    2553 generic.go:356] "Generic (PLEG): container finished" podID="42e4d112-aa98-4f8e-931a-487fcc4fa5c9" containerID="6f71b01469d42e32342947781d14d81b33e9ece28be25170bf696afa18e2e869" exitCode=0
Sep 05 02:20:41.990140 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.989843    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tfbs" event={"ID":"42e4d112-aa98-4f8e-931a-487fcc4fa5c9","Type":"ContainerDied","Data":"6f71b01469d42e32342947781d14d81b33e9ece28be25170bf696afa18e2e869"}
Sep 05 02:20:41.992240 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.992216    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5" event={"ID":"3266367a-da1d-42be-8f8f-b7d05cf1a9c4","Type":"ContainerStarted","Data":"47943e0298553d04a5c61b35c79687d821ca71077e14bcfa4ddf15ea1673dcbf"}
Sep 05 02:20:41.992347 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.992264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5" event={"ID":"3266367a-da1d-42be-8f8f-b7d05cf1a9c4","Type":"ContainerStarted","Data":"1f4f13fba9e73528f9c4f4f2fc7f1a91061371cf0816967e98bfcadb79d771eb"}
Sep 05 02:20:41.992347 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.992284    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5" event={"ID":"3266367a-da1d-42be-8f8f-b7d05cf1a9c4","Type":"ContainerStarted","Data":"cb06875c01fd521178412e0b8ac740f55f7a033df5c606ceec30a2e740d62138"}
Sep 05 02:20:41.994641 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.994619    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc" event={"ID":"82310afe-e173-4719-a2df-0333fb883f8e","Type":"ContainerStarted","Data":"8262c02f937ea6c17db63ef9b5ed593a91e8afd93afb07791640640bca47f8e5"}
Sep 05 02:20:41.996176 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.996151    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8" exitCode=0
Sep 05 02:20:41.996260 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:41.996185    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8"}
Sep 05 02:20:42.063646 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.063622    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:20:42.071005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.070962    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9d6fc" podStartSLOduration=2.867494774 podStartE2EDuration="4.070946204s" totalImagesPullingTime="1.20345143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:42.069539276 +0000 UTC m=+152.268902270" watchObservedRunningTime="2026-09-05 02:20:42.070946204 +0000 UTC m=+152.270309197"
Sep 05 02:20:42.098999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.098901    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"]
Sep 05 02:20:42.099402 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.099271    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gv6c5" podStartSLOduration=2.160399836 podStartE2EDuration="4.098805844s" totalImagesPullingTime="1.938406008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:42.097153393 +0000 UTC m=+152.296516415" watchObservedRunningTime="2026-09-05 02:20:42.098805844 +0000 UTC m=+152.298168838"
Sep 05 02:20:42.101802 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:42.101771    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510f85ce_e160_4340_b969_857c4c9a15e4.slice/crio-6103ff0818e26a2c00ee165e5a11555a25a41e6e831e9d0f184913f5ab33266a WatchSource:0}: Error finding container 6103ff0818e26a2c00ee165e5a11555a25a41e6e831e9d0f184913f5ab33266a: Status 404 returned error can't find the container with id 6103ff0818e26a2c00ee165e5a11555a25a41e6e831e9d0f184913f5ab33266a
Sep 05 02:20:42.347883 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.347860    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rvl98"]
Sep 05 02:20:42.350001 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:42.349977    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965dc0fc_5506_42dd_a079_0d548fdd31c4.slice/crio-f626d01f0889abfe4f6a25c245896d1f7acf275679a5c4bc9d5a842550339977 WatchSource:0}: Error finding container f626d01f0889abfe4f6a25c245896d1f7acf275679a5c4bc9d5a842550339977: Status 404 returned error can't find the container with id f626d01f0889abfe4f6a25c245896d1f7acf275679a5c4bc9d5a842550339977
Sep 05 02:20:42.824573 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.824529    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"]
Sep 05 02:20:42.827961 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.827943    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.830599 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.830574    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 05 02:20:42.830701 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.830596    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4mcg5\""
Sep 05 02:20:42.831924 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.831905    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 05 02:20:42.832113 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.832096    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 05 02:20:42.832175 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.832132    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 05 02:20:42.832175 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.832152    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9sim4qaffs25d\""
Sep 05 02:20:42.843554 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.843536    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"]
Sep 05 02:20:42.982463 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982431    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1142348b-19ce-4c4c-b173-176ca6ee6130-audit-log\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.982658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982532    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7mq\" (UniqueName: \"kubernetes.io/projected/1142348b-19ce-4c4c-b173-176ca6ee6130-kube-api-access-pn7mq\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.982658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982619    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-secret-metrics-server-client-certs\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.982658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982648    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1142348b-19ce-4c4c-b173-176ca6ee6130-metrics-server-audit-profiles\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.982842 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982674    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1142348b-19ce-4c4c-b173-176ca6ee6130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.982842 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-secret-metrics-server-tls\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:42.982842 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:42.982757    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-client-ca-bundle\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.004193 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.004151    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tfbs" event={"ID":"42e4d112-aa98-4f8e-931a-487fcc4fa5c9","Type":"ContainerStarted","Data":"77bf6c9ff25a56d2f190fca1319fc49f65e7310e276e90681eaa09e6b162f390"}
Sep 05 02:20:43.004592 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.004218    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tfbs" event={"ID":"42e4d112-aa98-4f8e-931a-487fcc4fa5c9","Type":"ContainerStarted","Data":"e45d442e5a11fe4c32558d1831898d2a83530ae1c364cacb928754bc98b7f941"}
Sep 05 02:20:43.005644 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.005590    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rvl98" event={"ID":"965dc0fc-5506-42dd-a079-0d548fdd31c4","Type":"ContainerStarted","Data":"f626d01f0889abfe4f6a25c245896d1f7acf275679a5c4bc9d5a842550339977"}
Sep 05 02:20:43.007073 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.007045    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"6103ff0818e26a2c00ee165e5a11555a25a41e6e831e9d0f184913f5ab33266a"}
Sep 05 02:20:43.028739 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.027936    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9tfbs" podStartSLOduration=3.423130157 podStartE2EDuration="5.027916676s" totalImagesPullingTime="1.53417398s" totalInitContainerRuntime="70.612539ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:43.02594795 +0000 UTC m=+153.225310941" watchObservedRunningTime="2026-09-05 02:20:43.027916676 +0000 UTC m=+153.227279672"
Sep 05 02:20:43.083934 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.083861    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1142348b-19ce-4c4c-b173-176ca6ee6130-audit-log\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084058 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.083953    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7mq\" (UniqueName: \"kubernetes.io/projected/1142348b-19ce-4c4c-b173-176ca6ee6130-kube-api-access-pn7mq\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084058 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.084008    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-secret-metrics-server-client-certs\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084058 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.084034    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1142348b-19ce-4c4c-b173-176ca6ee6130-metrics-server-audit-profiles\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084201 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.084065    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1142348b-19ce-4c4c-b173-176ca6ee6130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084201 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.084116    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-secret-metrics-server-tls\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084201 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.084150    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-client-ca-bundle\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.084643 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.084618    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1142348b-19ce-4c4c-b173-176ca6ee6130-audit-log\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.085443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.085200    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1142348b-19ce-4c4c-b173-176ca6ee6130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.085443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.085427    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1142348b-19ce-4c4c-b173-176ca6ee6130-metrics-server-audit-profiles\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.087900 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.087874    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-client-ca-bundle\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.088589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.088535    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-secret-metrics-server-client-certs\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.095281 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.095257    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7mq\" (UniqueName: \"kubernetes.io/projected/1142348b-19ce-4c4c-b173-176ca6ee6130-kube-api-access-pn7mq\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.103341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.103298    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1142348b-19ce-4c4c-b173-176ca6ee6130-secret-metrics-server-tls\") pod \"metrics-server-7bd5f8c8dd-4zkgm\" (UID: \"1142348b-19ce-4c4c-b173-176ca6ee6130\") " pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.137813 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.137583    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:20:43.239008 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.237789    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"]
Sep 05 02:20:43.243123 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.242677    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.249229 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.249204    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 05 02:20:43.249542 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.249522    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w77xs\""
Sep 05 02:20:43.250845 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.250800    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"]
Sep 05 02:20:43.387302 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.387223    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wmv\" (UniqueName: \"kubernetes.io/projected/27cecc77-d095-4df3-bc52-29b54fd193d0-kube-api-access-k7wmv\") pod \"monitoring-plugin-85bbbbbf4-xcm7r\" (UID: \"27cecc77-d095-4df3-bc52-29b54fd193d0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.387302 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.387289    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27cecc77-d095-4df3-bc52-29b54fd193d0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-xcm7r\" (UID: \"27cecc77-d095-4df3-bc52-29b54fd193d0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.446359 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.446283    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"]
Sep 05 02:20:43.450207 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:43.450169    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1142348b_19ce_4c4c_b173_176ca6ee6130.slice/crio-1e54d27bc1d4f5928bc82e515d531ae5377349911ffcc6062b6d4f9bea858dae WatchSource:0}: Error finding container 1e54d27bc1d4f5928bc82e515d531ae5377349911ffcc6062b6d4f9bea858dae: Status 404 returned error can't find the container with id 1e54d27bc1d4f5928bc82e515d531ae5377349911ffcc6062b6d4f9bea858dae
Sep 05 02:20:43.488799 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.488602    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wmv\" (UniqueName: \"kubernetes.io/projected/27cecc77-d095-4df3-bc52-29b54fd193d0-kube-api-access-k7wmv\") pod \"monitoring-plugin-85bbbbbf4-xcm7r\" (UID: \"27cecc77-d095-4df3-bc52-29b54fd193d0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.488799 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.488651    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27cecc77-d095-4df3-bc52-29b54fd193d0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-xcm7r\" (UID: \"27cecc77-d095-4df3-bc52-29b54fd193d0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.492066 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.491968    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27cecc77-d095-4df3-bc52-29b54fd193d0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-xcm7r\" (UID: \"27cecc77-d095-4df3-bc52-29b54fd193d0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.497445 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.497420    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wmv\" (UniqueName: \"kubernetes.io/projected/27cecc77-d095-4df3-bc52-29b54fd193d0-kube-api-access-k7wmv\") pod \"monitoring-plugin-85bbbbbf4-xcm7r\" (UID: \"27cecc77-d095-4df3-bc52-29b54fd193d0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:43.560869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:43.560840    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:44.013524 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.013450    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm" event={"ID":"1142348b-19ce-4c4c-b173-176ca6ee6130","Type":"ContainerStarted","Data":"1e54d27bc1d4f5928bc82e515d531ae5377349911ffcc6062b6d4f9bea858dae"}
Sep 05 02:20:44.683909 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.683876    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:20:44.702785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.702760    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:20:44.702959 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.702936    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.705653 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.705432    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.706234    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3adg565164sbp\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.706458    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.706948    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.707129    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.707291    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.707464    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.707651    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.707731    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.708083    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.708267    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.708600    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2sbts\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.708798    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 05 02:20:44.709220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.709099    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 05 02:20:44.712732 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.712539    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 05 02:20:44.716836 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.716787    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 05 02:20:44.805668 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805632    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805769 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805679    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805769 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805708    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805769 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805734    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805769    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805793    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74kw\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-kube-api-access-b74kw\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805820    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805882    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.805942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805929    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.805972    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806000    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-config\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806027    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806053    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806097    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806135    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806159    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806189    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806235    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.806762 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.806262    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907600 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907554    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907699 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907699 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907669    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907720    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907745    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907774    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.907811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908035 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907835    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908035 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907859    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74kw\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-kube-api-access-b74kw\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908035 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907892    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908035 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907915    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908035 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.907956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908035 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908008    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908051    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908080    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-config\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908106    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908135    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908185    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908227    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908253    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.908680 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.908662    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.912785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.912423    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.912785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.912431    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.912785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.912755    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.913869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.913037    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.913869 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.913512    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.916050 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.914787    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.920836 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.919523    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.922958 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.922873    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.923982    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-config\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924053    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924468    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924480    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924482    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74kw\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-kube-api-access-b74kw\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924480    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924976 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924911    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.924976 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.924949    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.925259 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.925211    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.930783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.928065    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.930783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.929736    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:44.958431 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:44.958357    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"]
Sep 05 02:20:44.963798 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:44.963769    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cecc77_d095_4df3_bc52_29b54fd193d0.slice/crio-b4e86dcbe89eb3fef7e0e8d0ab217c4ecf75fe6c1bbf6f4dada0bf3f1c1b672a WatchSource:0}: Error finding container b4e86dcbe89eb3fef7e0e8d0ab217c4ecf75fe6c1bbf6f4dada0bf3f1c1b672a: Status 404 returned error can't find the container with id b4e86dcbe89eb3fef7e0e8d0ab217c4ecf75fe6c1bbf6f4dada0bf3f1c1b672a
Sep 05 02:20:45.026209 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.026180    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"c83184de2bf972970dd6e80b188dfef5417269b2c6aff8a1bf5b45e83ef5f5f9"}
Sep 05 02:20:45.026663 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.026294    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"8b68cf1e426669d955c1effb3f9147f7775ec9c2b1e30628d58e4162356e4665"}
Sep 05 02:20:45.026663 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.026403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"7da4952736813e157f72ec9e4300921660f4f50a19d6043ec4fda528542b6595"}
Sep 05 02:20:45.030417 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.030208    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9"}
Sep 05 02:20:45.030417 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.030280    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf"}
Sep 05 02:20:45.030417 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.030323    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a"}
Sep 05 02:20:45.034201 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.034174    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r" event={"ID":"27cecc77-d095-4df3-bc52-29b54fd193d0","Type":"ContainerStarted","Data":"b4e86dcbe89eb3fef7e0e8d0ab217c4ecf75fe6c1bbf6f4dada0bf3f1c1b672a"}
Sep 05 02:20:45.049528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.049507    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:20:45.353655 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:45.353539    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:20:45.355131 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:45.355101    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626aa754_77ce_4648_8fa0_5ff95c6338f6.slice/crio-3d2d3bef2892a3302e05144e8117a4a14a675f9ec9fe28f5cdece65d285c0768 WatchSource:0}: Error finding container 3d2d3bef2892a3302e05144e8117a4a14a675f9ec9fe28f5cdece65d285c0768: Status 404 returned error can't find the container with id 3d2d3bef2892a3302e05144e8117a4a14a675f9ec9fe28f5cdece65d285c0768
Sep 05 02:20:46.042133 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:46.042095    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830" exitCode=0
Sep 05 02:20:46.042552 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:46.042180    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"}
Sep 05 02:20:46.042552 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:46.042268    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"3d2d3bef2892a3302e05144e8117a4a14a675f9ec9fe28f5cdece65d285c0768"}
Sep 05 02:20:46.047899 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:46.047874    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff"}
Sep 05 02:20:46.048009 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:46.047943    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3"}
Sep 05 02:20:47.058554 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.058512    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"5d93ef891206fdfd0e8523f75fea4f550ff33c9074409f989ed28947e3a16b4b"}
Sep 05 02:20:47.059052 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.058616    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"b44ceb0365b9cd355f7ebd0fc42b68b88bafd1b01f32f09febe16e260c47c875"}
Sep 05 02:20:47.059052 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.058660    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" event={"ID":"510f85ce-e160-4340-b969-857c4c9a15e4","Type":"ContainerStarted","Data":"92239bbd353154fcd67231879a40c45b24e84be11da4f2d7348830c28c276fa7"}
Sep 05 02:20:47.059848 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.059752    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:47.062542 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.062006    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm" event={"ID":"1142348b-19ce-4c4c-b173-176ca6ee6130","Type":"ContainerStarted","Data":"587e62ffd626f628e40a022c1b70888201024124d0ea347b1fc78ba25f5bfd39"}
Sep 05 02:20:47.069275 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.068325    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerStarted","Data":"273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52"}
Sep 05 02:20:47.091605 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.090030    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn" podStartSLOduration=2.318660596 podStartE2EDuration="6.090010604s" totalImagesPullingTime="3.771350008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:47.088540185 +0000 UTC m=+157.287903183" watchObservedRunningTime="2026-09-05 02:20:47.090010604 +0000 UTC m=+157.289373596"
Sep 05 02:20:47.120620 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:47.120543    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.893215908 podStartE2EDuration="8.120522644s" totalImagesPullingTime="5.1997448s" totalInitContainerRuntime="27.561936ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:47.118975135 +0000 UTC m=+157.318338129" watchObservedRunningTime="2026-09-05 02:20:47.120522644 +0000 UTC m=+157.319885639"
Sep 05 02:20:47.721799 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:47.721711    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac"
Sep 05 02:20:47.733217 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:20:47.733168    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pcsnn" podUID="b72ef229-be3c-4e6e-8d1f-59488d8a46ff"
Sep 05 02:20:48.076027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.075905    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r" event={"ID":"27cecc77-d095-4df3-bc52-29b54fd193d0","Type":"ContainerStarted","Data":"4747b0d4e68acec1bca00e58c58caa499f990afcbe0a0eea18925ce865c65705"}
Sep 05 02:20:48.076470 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.076329    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:20:48.076470 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.076342    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:20:48.077092 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.077069    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:48.082762 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.082708    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r"
Sep 05 02:20:48.093888 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.093809    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-xcm7r" podStartSLOduration=2.67032378 podStartE2EDuration="5.093795002s" totalImagesPullingTime="2.423471222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:48.092767489 +0000 UTC m=+158.292130488" watchObservedRunningTime="2026-09-05 02:20:48.093795002 +0000 UTC m=+158.293157998"
Sep 05 02:20:48.095435 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:48.095391    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm" podStartSLOduration=3.34382451 podStartE2EDuration="6.095377534s" totalImagesPullingTime="2.751553024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:47.144922206 +0000 UTC m=+157.344285199" watchObservedRunningTime="2026-09-05 02:20:48.095377534 +0000 UTC m=+158.294740529"
Sep 05 02:20:49.089290 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:49.089252    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64bfc7bdcb-nqvfn"
Sep 05 02:20:50.092057 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:50.092034    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"}
Sep 05 02:20:50.092413 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:50.092114    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"}
Sep 05 02:20:51.100807 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:51.100772    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"}
Sep 05 02:20:51.101269 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:51.100836    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"}
Sep 05 02:20:51.101269 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:51.100863    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"}
Sep 05 02:20:51.101269 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:51.100901    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerStarted","Data":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"}
Sep 05 02:20:51.136111 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:51.135951    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.251258247 podStartE2EDuration="7.135931838s" totalImagesPullingTime="3.860068159s" totalInitContainerRuntime="24.605432ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:51.13328323 +0000 UTC m=+161.332646238" watchObservedRunningTime="2026-09-05 02:20:51.135931838 +0000 UTC m=+161.335294829"
Sep 05 02:20:52.589534 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.589486    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:20:52.589952 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.589594    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:20:52.591747 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.591727    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b72ef229-be3c-4e6e-8d1f-59488d8a46ff-metrics-tls\") pod \"dns-default-pcsnn\" (UID: \"b72ef229-be3c-4e6e-8d1f-59488d8a46ff\") " pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:20:52.591931 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.591913    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"image-registry-579dd5547b-2vc4k\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") " pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:20:52.881452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.881363    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9hcm\""
Sep 05 02:20:52.881452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.881363    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hnv4q\""
Sep 05 02:20:52.888024 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.888004    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:20:52.888150 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:52.888094    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:20:53.171335 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:53.171310    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcsnn"]
Sep 05 02:20:53.173610 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:53.173580    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72ef229_be3c_4e6e_8d1f_59488d8a46ff.slice/crio-5709b5a8c464e2371df42fc7ba9847264d03d595d133e9f88f9acc868f1fe040 WatchSource:0}: Error finding container 5709b5a8c464e2371df42fc7ba9847264d03d595d133e9f88f9acc868f1fe040: Status 404 returned error can't find the container with id 5709b5a8c464e2371df42fc7ba9847264d03d595d133e9f88f9acc868f1fe040
Sep 05 02:20:53.180899 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:53.180743    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579dd5547b-2vc4k"]
Sep 05 02:20:53.182866 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:20:53.182840    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ee1e15_56a4_4c32_9984_6545d50257ac.slice/crio-423476e49c4851c284fe4cdb1a2b3310ea0a224275af5f4d59074844fed88217 WatchSource:0}: Error finding container 423476e49c4851c284fe4cdb1a2b3310ea0a224275af5f4d59074844fed88217: Status 404 returned error can't find the container with id 423476e49c4851c284fe4cdb1a2b3310ea0a224275af5f4d59074844fed88217
Sep 05 02:20:54.120951 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:54.120909    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" event={"ID":"a0ee1e15-56a4-4c32-9984-6545d50257ac","Type":"ContainerStarted","Data":"af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821"}
Sep 05 02:20:54.121377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:54.121012    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:20:54.121377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:54.121037    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" event={"ID":"a0ee1e15-56a4-4c32-9984-6545d50257ac","Type":"ContainerStarted","Data":"423476e49c4851c284fe4cdb1a2b3310ea0a224275af5f4d59074844fed88217"}
Sep 05 02:20:54.122311 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:54.122285    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcsnn" event={"ID":"b72ef229-be3c-4e6e-8d1f-59488d8a46ff","Type":"ContainerStarted","Data":"5709b5a8c464e2371df42fc7ba9847264d03d595d133e9f88f9acc868f1fe040"}
Sep 05 02:20:54.146332 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:54.146280    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" podStartSLOduration=163.146262644 podStartE2EDuration="2m43.146262644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:54.145943277 +0000 UTC m=+164.345306269" watchObservedRunningTime="2026-09-05 02:20:54.146262644 +0000 UTC m=+164.345625636"
Sep 05 02:20:55.049920 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:20:55.049885    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:01.160820 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:01.160758    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rvl98" event={"ID":"965dc0fc-5506-42dd-a079-0d548fdd31c4","Type":"ContainerStarted","Data":"5351b208b5887ba3f6e083d63c79de59d4879b2d55dbabc91dc8542323eafa48"}
Sep 05 02:21:01.161693 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:01.161669    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:21:01.180328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:01.180292    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-rvl98"
Sep 05 02:21:01.182101 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:01.181981    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-rvl98" podStartSLOduration=2.063559371 podStartE2EDuration="20.181961809s" totalImagesPullingTime="18.118402438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:01.180094442 +0000 UTC m=+171.379457436" watchObservedRunningTime="2026-09-05 02:21:01.181961809 +0000 UTC m=+171.381324802"
Sep 05 02:21:03.138117 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:03.138069    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:21:03.138555 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:03.138179    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:21:03.172690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:03.172651    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcsnn" event={"ID":"b72ef229-be3c-4e6e-8d1f-59488d8a46ff","Type":"ContainerStarted","Data":"c87aaaaf5728293d2e895dc1f17447da5cf0d97ed07bfe6b02bbe25c2a30181d"}
Sep 05 02:21:03.172690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:03.172720    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcsnn" event={"ID":"b72ef229-be3c-4e6e-8d1f-59488d8a46ff","Type":"ContainerStarted","Data":"48d46df8bcb4e75d3742790fb15099d2bc5037069b5b6fa9ebca40ef0904e26e"}
Sep 05 02:21:03.189721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:03.189658    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pcsnn" podStartSLOduration=129.883737423 podStartE2EDuration="2m19.189625913s" totalImagesPullingTime="9.30588849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:03.189389527 +0000 UTC m=+173.388752545" watchObservedRunningTime="2026-09-05 02:21:03.189625913 +0000 UTC m=+173.388989108"
Sep 05 02:21:04.177824 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:04.177785    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:21:12.892272 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:12.892236    2553 patch_prober.go:28] interesting pod/image-registry-579dd5547b-2vc4k container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:21:12.892794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:12.892288    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:21:14.184459 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:14.184431    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcsnn"
Sep 05 02:21:15.132574 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:15.132529    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:21:16.695659 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:16.695615    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/init-config-reloader/0.log"
Sep 05 02:21:16.895732 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:16.895703    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/alertmanager/0.log"
Sep 05 02:21:17.096237 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:17.096197    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/config-reloader/0.log"
Sep 05 02:21:17.304367 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:17.304336    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/kube-rbac-proxy-web/0.log"
Sep 05 02:21:17.497089 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:17.496969    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/kube-rbac-proxy/0.log"
Sep 05 02:21:17.695491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:17.695466    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/kube-rbac-proxy-metric/0.log"
Sep 05 02:21:17.895376 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:17.895350    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_adfda1ed-6c0a-4702-9992-7466df7421a5/prom-label-proxy/0.log"
Sep 05 02:21:18.295305 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:18.295277    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gv6c5_3266367a-da1d-42be-8f8f-b7d05cf1a9c4/kube-state-metrics/0.log"
Sep 05 02:21:18.496989 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:18.496961    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gv6c5_3266367a-da1d-42be-8f8f-b7d05cf1a9c4/kube-rbac-proxy-main/0.log"
Sep 05 02:21:18.696385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:18.696285    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gv6c5_3266367a-da1d-42be-8f8f-b7d05cf1a9c4/kube-rbac-proxy-self/0.log"
Sep 05 02:21:18.895057 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:18.895032    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd5f8c8dd-4zkgm_1142348b-19ce-4c4c-b173-176ca6ee6130/metrics-server/0.log"
Sep 05 02:21:19.095134 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:19.095109    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-xcm7r_27cecc77-d095-4df3-bc52-29b54fd193d0/monitoring-plugin/0.log"
Sep 05 02:21:19.295177 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:19.295150    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tfbs_42e4d112-aa98-4f8e-931a-487fcc4fa5c9/init-textfile/0.log"
Sep 05 02:21:19.496226 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:19.496170    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tfbs_42e4d112-aa98-4f8e-931a-487fcc4fa5c9/node-exporter/0.log"
Sep 05 02:21:19.698023 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:19.697997    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tfbs_42e4d112-aa98-4f8e-931a-487fcc4fa5c9/kube-rbac-proxy/0.log"
Sep 05 02:21:21.094974 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:21.094942    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9d6fc_82310afe-e173-4719-a2df-0333fb883f8e/kube-rbac-proxy-main/0.log"
Sep 05 02:21:21.296277 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:21.296224    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9d6fc_82310afe-e173-4719-a2df-0333fb883f8e/kube-rbac-proxy-self/0.log"
Sep 05 02:21:21.495513 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:21.495440    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9d6fc_82310afe-e173-4719-a2df-0333fb883f8e/openshift-state-metrics/0.log"
Sep 05 02:21:21.696118 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:21.696091    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/init-config-reloader/0.log"
Sep 05 02:21:21.897188 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:21.897162    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/prometheus/0.log"
Sep 05 02:21:22.101493 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:22.101470    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/config-reloader/0.log"
Sep 05 02:21:22.295756 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:22.295734    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/thanos-sidecar/0.log"
Sep 05 02:21:22.495447 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:22.495425    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/kube-rbac-proxy-web/0.log"
Sep 05 02:21:22.695709 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:22.695641    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/kube-rbac-proxy/0.log"
Sep 05 02:21:22.900286 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:22.900256    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_626aa754-77ce-4648-8fa0-5ff95c6338f6/kube-rbac-proxy-thanos/0.log"
Sep 05 02:21:23.099407 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.099384    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-mq44t_a5977287-5f4c-4ea2-b4e8-424f7f329c4c/prometheus-operator/0.log"
Sep 05 02:21:23.143826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.143805    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:21:23.147622 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.147602    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bd5f8c8dd-4zkgm"
Sep 05 02:21:23.298268 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.298235    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-mq44t_a5977287-5f4c-4ea2-b4e8-424f7f329c4c/kube-rbac-proxy/0.log"
Sep 05 02:21:23.495409 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.495346    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-zk6kc_ea0af986-d6ca-4e86-b651-180096161852/prometheus-operator-admission-webhook/0.log"
Sep 05 02:21:23.697005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.696976    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/thanos-query/0.log"
Sep 05 02:21:23.899953 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:23.899929    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy-web/0.log"
Sep 05 02:21:24.098022 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:24.097991    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy/0.log"
Sep 05 02:21:24.295407 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:24.295382    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/prom-label-proxy/0.log"
Sep 05 02:21:24.497513 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:24.497489    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy-rules/0.log"
Sep 05 02:21:24.699700 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:24.699608    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy-metrics/0.log"
Sep 05 02:21:24.896093 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:24.896065    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-qvr68_6dd08154-87bf-435e-b2cc-827a2419b85f/networking-console-plugin/0.log"
Sep 05 02:21:24.917770 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:24.917742    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579dd5547b-2vc4k"]
Sep 05 02:21:25.896673 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:25.896641    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-rvl98_965dc0fc-5506-42dd-a079-0d548fdd31c4/download-server/0.log"
Sep 05 02:21:26.495340 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:26.495303    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcsnn_b72ef229-be3c-4e6e-8d1f-59488d8a46ff/dns/0.log"
Sep 05 02:21:26.695680 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:26.695652    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcsnn_b72ef229-be3c-4e6e-8d1f-59488d8a46ff/kube-rbac-proxy/0.log"
Sep 05 02:21:27.496517 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:27.496492    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-96gdl_0d380942-90b8-482d-a917-d4d3435fb445/dns-node-resolver/0.log"
Sep 05 02:21:45.050464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:45.050423    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:45.066854 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:45.066828    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:45.369656 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:45.369587    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:21:49.937151 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:49.937091    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac" containerName="registry" containerID="cri-o://af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821" gracePeriod=30
Sep 05 02:21:51.177759 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.177735    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:21:51.333260 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333229    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-installation-pull-secrets\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333282    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-certificates\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333309    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333338    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kprx\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-kube-api-access-5kprx\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333356    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-bound-sa-token\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333376    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-image-registry-private-configuration\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333413    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0ee1e15-56a4-4c32-9984-6545d50257ac-ca-trust-extracted\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333772 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333496    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-trusted-ca\") pod \"a0ee1e15-56a4-4c32-9984-6545d50257ac\" (UID: \"a0ee1e15-56a4-4c32-9984-6545d50257ac\") "
Sep 05 02:21:51.333772 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.333713    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:51.334206 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.334163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:51.335911 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.335865    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-kube-api-access-5kprx" (OuterVolumeSpecName: "kube-api-access-5kprx") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "kube-api-access-5kprx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:51.336025 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.335977    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:51.336083 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.336044    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:51.336174 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.336138    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:51.336322 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.336303    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:51.341982 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.341944    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ee1e15-56a4-4c32-9984-6545d50257ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a0ee1e15-56a4-4c32-9984-6545d50257ac" (UID: "a0ee1e15-56a4-4c32-9984-6545d50257ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:21:51.381691 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.381664    2553 generic.go:356] "Generic (PLEG): container finished" podID="a0ee1e15-56a4-4c32-9984-6545d50257ac" containerID="af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821" exitCode=0
Sep 05 02:21:51.381811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.381729    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k"
Sep 05 02:21:51.381811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.381753    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" event={"ID":"a0ee1e15-56a4-4c32-9984-6545d50257ac","Type":"ContainerDied","Data":"af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821"}
Sep 05 02:21:51.381880 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.381828    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579dd5547b-2vc4k" event={"ID":"a0ee1e15-56a4-4c32-9984-6545d50257ac","Type":"ContainerDied","Data":"423476e49c4851c284fe4cdb1a2b3310ea0a224275af5f4d59074844fed88217"}
Sep 05 02:21:51.381880 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.381849    2553 scope.go:117] "RemoveContainer" containerID="af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821"
Sep 05 02:21:51.390318 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.390301    2553 scope.go:117] "RemoveContainer" containerID="af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821"
Sep 05 02:21:51.390603 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:21:51.390583    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821\": container with ID starting with af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821 not found: ID does not exist" containerID="af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821"
Sep 05 02:21:51.390647 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.390613    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821"} err="failed to get container status \"af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821\": rpc error: code = NotFound desc = could not find container \"af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821\": container with ID starting with af974261f537ce324223d1c76fd85088bce2a0891cea71da11450075d4a49821 not found: ID does not exist"
Sep 05 02:21:51.402920 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.402895    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579dd5547b-2vc4k"]
Sep 05 02:21:51.406427 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.406408    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-579dd5547b-2vc4k"]
Sep 05 02:21:51.434165 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434145    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-trusted-ca\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434165 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434166    2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-installation-pull-secrets\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434175    2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-certificates\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434184    2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-registry-tls\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434193    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kprx\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-kube-api-access-5kprx\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434200    2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0ee1e15-56a4-4c32-9984-6545d50257ac-bound-sa-token\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434209    2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0ee1e15-56a4-4c32-9984-6545d50257ac-image-registry-private-configuration\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:51.434284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:51.434217    2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0ee1e15-56a4-4c32-9984-6545d50257ac-ca-trust-extracted\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:21:52.391040 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:52.391007    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac" path="/var/lib/kubelet/pods/a0ee1e15-56a4-4c32-9984-6545d50257ac/volumes"
Sep 05 02:21:58.665544 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.665369    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:58.666116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.665852    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="alertmanager" containerID="cri-o://2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a" gracePeriod=120
Sep 05 02:21:58.666116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.665942    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-metric" containerID="cri-o://970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff" gracePeriod=120
Sep 05 02:21:58.666116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.666048    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy" containerID="cri-o://614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3" gracePeriod=120
Sep 05 02:21:58.666295 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.666137    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-web" containerID="cri-o://775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9" gracePeriod=120
Sep 05 02:21:58.666295 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.666150    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="prom-label-proxy" containerID="cri-o://273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52" gracePeriod=120
Sep 05 02:21:58.666295 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:58.666234    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="config-reloader" containerID="cri-o://40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf" gracePeriod=120
Sep 05 02:21:59.415584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415540    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52" exitCode=0
Sep 05 02:21:59.415584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415579    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff" exitCode=0
Sep 05 02:21:59.415584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415587    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3" exitCode=0
Sep 05 02:21:59.415584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415595    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf" exitCode=0
Sep 05 02:21:59.415906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415601    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a" exitCode=0
Sep 05 02:21:59.415906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415606    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52"}
Sep 05 02:21:59.415906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415687    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff"}
Sep 05 02:21:59.415906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415730    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3"}
Sep 05 02:21:59.415906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415766    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf"}
Sep 05 02:21:59.415906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.415805    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a"}
Sep 05 02:21:59.916117 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:21:59.916092    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.004603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004513    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004551    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g628x\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-kube-api-access-g628x\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004591    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004617    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-web-config\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004636    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004670    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-config-out\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004707    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-trusted-ca-bundle\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004731    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-main-db\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004760    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-main-tls\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004811    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-config-volume\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004840    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-metrics-client-ca\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.004861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004864    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-tls-assets\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.005373 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.004887    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-cluster-tls-config\") pod \"adfda1ed-6c0a-4702-9992-7466df7421a5\" (UID: \"adfda1ed-6c0a-4702-9992-7466df7421a5\") "
Sep 05 02:22:00.005431 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.005361    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:00.005870 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.005814    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:00.006089 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.005988    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:22:00.007947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.007908    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-kube-api-access-g628x" (OuterVolumeSpecName: "kube-api-access-g628x") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "kube-api-access-g628x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:22:00.008060 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.008016    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.008151 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.008121    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-config-out" (OuterVolumeSpecName: "config-out") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:22:00.008277 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.008251    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-config-volume" (OuterVolumeSpecName: "config-volume") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.008371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.008348    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.008432 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.008364    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.008960 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.008936    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:22:00.009495 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.009471    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.012929 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.012906    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.018970 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.018945    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-web-config" (OuterVolumeSpecName: "web-config") pod "adfda1ed-6c0a-4702-9992-7466df7421a5" (UID: "adfda1ed-6c0a-4702-9992-7466df7421a5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:00.106409 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106380    2553 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-config-volume\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106409 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106406    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-metrics-client-ca\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106420    2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-tls-assets\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106429    2553 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-cluster-tls-config\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106438    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106447    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g628x\" (UniqueName: \"kubernetes.io/projected/adfda1ed-6c0a-4702-9992-7466df7421a5-kube-api-access-g628x\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106455    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106464    2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-web-config\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106472    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106480    2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-config-out\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106494    2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106503    2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/adfda1ed-6c0a-4702-9992-7466df7421a5-alertmanager-main-db\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.106589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.106512    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/adfda1ed-6c0a-4702-9992-7466df7421a5-secret-alertmanager-main-tls\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:00.422905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.422872    2553 generic.go:356] "Generic (PLEG): container finished" podID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerID="775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9" exitCode=0
Sep 05 02:22:00.423045 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.422912    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9"}
Sep 05 02:22:00.423045 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.422946    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.423045 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.422965    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"adfda1ed-6c0a-4702-9992-7466df7421a5","Type":"ContainerDied","Data":"a409b0191c21a26662fd246c58de7d2946fba30390624576a9e26c574d29db91"}
Sep 05 02:22:00.423045 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.422999    2553 scope.go:117] "RemoveContainer" containerID="273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52"
Sep 05 02:22:00.431199 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.431178    2553 scope.go:117] "RemoveContainer" containerID="970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff"
Sep 05 02:22:00.438085 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.438068    2553 scope.go:117] "RemoveContainer" containerID="614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3"
Sep 05 02:22:00.445593 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.445549    2553 scope.go:117] "RemoveContainer" containerID="775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9"
Sep 05 02:22:00.446256 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.446235    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:22:00.451658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.451634    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:22:00.453031 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.453017    2553 scope.go:117] "RemoveContainer" containerID="40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf"
Sep 05 02:22:00.459787 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.459772    2553 scope.go:117] "RemoveContainer" containerID="2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a"
Sep 05 02:22:00.466716 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.466697    2553 scope.go:117] "RemoveContainer" containerID="16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8"
Sep 05 02:22:00.473857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.473840    2553 scope.go:117] "RemoveContainer" containerID="273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52"
Sep 05 02:22:00.474163 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.474131    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52\": container with ID starting with 273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52 not found: ID does not exist" containerID="273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52"
Sep 05 02:22:00.474211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.474174    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52"} err="failed to get container status \"273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52\": rpc error: code = NotFound desc = could not find container \"273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52\": container with ID starting with 273ab6c9cbd8855a5365c0e23ee8920d0cb2bb1766ea980cefbe8a3100798a52 not found: ID does not exist"
Sep 05 02:22:00.474211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.474193    2553 scope.go:117] "RemoveContainer" containerID="970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff"
Sep 05 02:22:00.474463 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.474444    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff\": container with ID starting with 970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff not found: ID does not exist" containerID="970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff"
Sep 05 02:22:00.474504 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.474470    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff"} err="failed to get container status \"970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff\": rpc error: code = NotFound desc = could not find container \"970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff\": container with ID starting with 970d0c08005f0589f4968b26e382a074ea5c65471ded8a68a6b470752bf53dff not found: ID does not exist"
Sep 05 02:22:00.474504 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.474486    2553 scope.go:117] "RemoveContainer" containerID="614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3"
Sep 05 02:22:00.474762 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.474744    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3\": container with ID starting with 614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3 not found: ID does not exist" containerID="614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3"
Sep 05 02:22:00.474826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.474766    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3"} err="failed to get container status \"614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3\": rpc error: code = NotFound desc = could not find container \"614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3\": container with ID starting with 614c41d8e4b38e0325f7d6883d8df6dff2f9de78977a8de415cb4f7bdb331bb3 not found: ID does not exist"
Sep 05 02:22:00.474826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.474788    2553 scope.go:117] "RemoveContainer" containerID="775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9"
Sep 05 02:22:00.475013 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.474998    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9\": container with ID starting with 775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9 not found: ID does not exist" containerID="775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9"
Sep 05 02:22:00.475052 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475016    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9"} err="failed to get container status \"775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9\": rpc error: code = NotFound desc = could not find container \"775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9\": container with ID starting with 775de69fcd866ae8548b46a7f4d99a3c2a8824b20c93153e9babc593e6674ca9 not found: ID does not exist"
Sep 05 02:22:00.475052 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475028    2553 scope.go:117] "RemoveContainer" containerID="40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf"
Sep 05 02:22:00.475221 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.475209    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf\": container with ID starting with 40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf not found: ID does not exist" containerID="40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf"
Sep 05 02:22:00.475254 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475223    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf"} err="failed to get container status \"40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf\": rpc error: code = NotFound desc = could not find container \"40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf\": container with ID starting with 40743bac7f8be7ac6cdd5106904305603e46a9783808d1e775c9163d9563cfaf not found: ID does not exist"
Sep 05 02:22:00.475254 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475233    2553 scope.go:117] "RemoveContainer" containerID="2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a"
Sep 05 02:22:00.475444 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.475430    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a\": container with ID starting with 2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a not found: ID does not exist" containerID="2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a"
Sep 05 02:22:00.475488 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475445    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a"} err="failed to get container status \"2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a\": rpc error: code = NotFound desc = could not find container \"2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a\": container with ID starting with 2d9fffce4abdc771c8d573d3bfee280b5c3ff26177b98ddff81dc7e4fb017f4a not found: ID does not exist"
Sep 05 02:22:00.475488 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475455    2553 scope.go:117] "RemoveContainer" containerID="16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8"
Sep 05 02:22:00.475664 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:00.475646    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8\": container with ID starting with 16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8 not found: ID does not exist" containerID="16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8"
Sep 05 02:22:00.475713 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.475668    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8"} err="failed to get container status \"16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8\": rpc error: code = NotFound desc = could not find container \"16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8\": container with ID starting with 16a18bb1bb42fe10db829db3a248fbe90c89b1c85c36d6e6417e992c2ffbdcb8 not found: ID does not exist"
Sep 05 02:22:00.479508 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479488    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:22:00.479837 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479824    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="init-config-reloader"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479840    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="init-config-reloader"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479851    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="alertmanager"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479860    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="alertmanager"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479873    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-web"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479879    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-web"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479889    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-metric"
Sep 05 02:22:00.479893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479895    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-metric"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479907    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479912    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479920    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac" containerName="registry"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479927    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac" containerName="registry"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479937    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="config-reloader"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479943    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="config-reloader"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479949    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="prom-label-proxy"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479954    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="prom-label-proxy"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.479996    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-web"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.480006    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="alertmanager"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.480016    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy-metric"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.480023    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ee1e15-56a4-4c32-9984-6545d50257ac" containerName="registry"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.480028    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="config-reloader"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.480032    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="kube-rbac-proxy"
Sep 05 02:22:00.480141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.480038    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" containerName="prom-label-proxy"
Sep 05 02:22:00.485265 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.485246    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.488118 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488100    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 02:22:00.488258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488240    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 02:22:00.488333 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488249    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 02:22:00.488333 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488254    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 02:22:00.488484 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488466    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 02:22:00.488484 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488477    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 02:22:00.488639 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.488476    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 02:22:00.489448 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.489032    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pwrbs\""
Sep 05 02:22:00.489448 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.489049    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 02:22:00.499864 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.499844    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 02:22:00.499940 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.499915    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:22:00.610522 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610489    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610526    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610549    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-config-volume\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610618    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/731a132e-6ee1-4616-ac94-52dd63655ef4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610649    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610728    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/731a132e-6ee1-4616-ac94-52dd63655ef4-config-out\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610763    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610788    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-web-config\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610832 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610804    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/731a132e-6ee1-4616-ac94-52dd63655ef4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610974 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610854    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731a132e-6ee1-4616-ac94-52dd63655ef4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610974 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610915    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcss\" (UniqueName: \"kubernetes.io/projected/731a132e-6ee1-4616-ac94-52dd63655ef4-kube-api-access-6qcss\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.610974 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.610931    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/731a132e-6ee1-4616-ac94-52dd63655ef4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711353 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711274    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711353 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711311    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/731a132e-6ee1-4616-ac94-52dd63655ef4-config-out\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711353 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711334    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711360    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-web-config\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711398    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/731a132e-6ee1-4616-ac94-52dd63655ef4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711427    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731a132e-6ee1-4616-ac94-52dd63655ef4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711471    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcss\" (UniqueName: \"kubernetes.io/projected/731a132e-6ee1-4616-ac94-52dd63655ef4-kube-api-access-6qcss\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711495    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/731a132e-6ee1-4616-ac94-52dd63655ef4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711521    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711590    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.711637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711623    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-config-volume\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.712044 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711659    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/731a132e-6ee1-4616-ac94-52dd63655ef4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.712044 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.711692    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.712380 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.712344    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/731a132e-6ee1-4616-ac94-52dd63655ef4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.712829 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.712469    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731a132e-6ee1-4616-ac94-52dd63655ef4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.712829 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.712645    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/731a132e-6ee1-4616-ac94-52dd63655ef4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.714247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.714222    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/731a132e-6ee1-4616-ac94-52dd63655ef4-config-out\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.714395 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.714368    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.714551 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.714525    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.714681 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.714665    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.714737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.714679    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.714795 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.714766    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-web-config\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.715172 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.715148    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.715267 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.715174    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/731a132e-6ee1-4616-ac94-52dd63655ef4-config-volume\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.716232 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.716213    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/731a132e-6ee1-4616-ac94-52dd63655ef4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.719576 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.719544    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcss\" (UniqueName: \"kubernetes.io/projected/731a132e-6ee1-4616-ac94-52dd63655ef4-kube-api-access-6qcss\") pod \"alertmanager-main-0\" (UID: \"731a132e-6ee1-4616-ac94-52dd63655ef4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:00.799667 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:00.799624    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:22:01.082615 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:01.082580    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:22:01.083835 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:22:01.083803    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731a132e_6ee1_4616_ac94_52dd63655ef4.slice/crio-c568d85e1f867cc191f353bb5e34702560d0b40498a44d7e4a513c631a779e90 WatchSource:0}: Error finding container c568d85e1f867cc191f353bb5e34702560d0b40498a44d7e4a513c631a779e90: Status 404 returned error can't find the container with id c568d85e1f867cc191f353bb5e34702560d0b40498a44d7e4a513c631a779e90
Sep 05 02:22:01.429589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:01.429481    2553 generic.go:356] "Generic (PLEG): container finished" podID="731a132e-6ee1-4616-ac94-52dd63655ef4" containerID="ea07ac996b3d1ee20962a1fd7058c8abec0ac8ef8094f8f309c43c221648aaf9" exitCode=0
Sep 05 02:22:01.429721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:01.429585    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerDied","Data":"ea07ac996b3d1ee20962a1fd7058c8abec0ac8ef8094f8f309c43c221648aaf9"}
Sep 05 02:22:01.429721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:01.429652    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"c568d85e1f867cc191f353bb5e34702560d0b40498a44d7e4a513c631a779e90"}
Sep 05 02:22:02.392135 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.392104    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adfda1ed-6c0a-4702-9992-7466df7421a5" path="/var/lib/kubelet/pods/adfda1ed-6c0a-4702-9992-7466df7421a5/volumes"
Sep 05 02:22:02.436274 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.436245    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"55d24bf20463463ac188ecdd17408e9093eeaecf4defd7e601ce56bdcc0dada2"}
Sep 05 02:22:02.436381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.436304    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"539a1f6d8af1ba0a88a2faedf191b70681ab8d0cb3dc0b9a47b9d69fa79be60e"}
Sep 05 02:22:02.436381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.436332    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"88f7f286a540d7c4c7dfd5254e3edba5b57d858343819f20480937ece8508535"}
Sep 05 02:22:02.436381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.436357    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"da3c081163c5aa44a6d156d5e19a3f1bf08288faf0b7d3b6762e0f3cf8163250"}
Sep 05 02:22:02.436381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.436380    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"cac38ffc4caad88a7f7bb245435a0cf5a61a861ea3ba466a087a9c86b932354f"}
Sep 05 02:22:02.436601 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.436403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"731a132e-6ee1-4616-ac94-52dd63655ef4","Type":"ContainerStarted","Data":"58bfdac0959b218e295e94551c8e399179a4c12b03b565b8a088cb37c8d81a3f"}
Sep 05 02:22:02.469785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.469733    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.453521378 podStartE2EDuration="2.469712045s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.190667ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:22:02.467089614 +0000 UTC m=+232.666452622" watchObservedRunningTime="2026-09-05 02:22:02.469712045 +0000 UTC m=+232.669075039"
Sep 05 02:22:02.967972 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.967919    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:22:02.968381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.968324    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="prometheus" containerID="cri-o://eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50" gracePeriod=600
Sep 05 02:22:02.968504 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.968364    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc" gracePeriod=600
Sep 05 02:22:02.968504 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.968385    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-web" containerID="cri-o://9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0" gracePeriod=600
Sep 05 02:22:02.968504 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.968448    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="thanos-sidecar" containerID="cri-o://c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f" gracePeriod=600
Sep 05 02:22:02.968504 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.968469    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="config-reloader" containerID="cri-o://9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a" gracePeriod=600
Sep 05 02:22:02.968743 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:02.968473    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy" containerID="cri-o://69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8" gracePeriod=600
Sep 05 02:22:03.219647 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.219586    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.334727 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.334692    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-grpc-tls\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.334727 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.334727    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-tls-assets\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.334980 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.334755    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-tls\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.334980 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.334785    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-kube-rbac-proxy\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.334980 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.334960    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-metrics-client-certs\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335135 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335049    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-web-config\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335135 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335092    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335135 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335132    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-db\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335287 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335170    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-thanos-prometheus-http-client-file\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335287 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335208    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-1\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335287 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335261    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b74kw\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-kube-api-access-b74kw\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.335428 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335299    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-serving-certs-ca-bundle\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335616    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335820    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-metrics-client-ca\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335879    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-2\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335905    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335912    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-0\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335957    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-config-out\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.335991    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-config\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.336018    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-kubelet-serving-ca-bundle\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.336044    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-trusted-ca-bundle\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.336079    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"626aa754-77ce-4648-8fa0-5ff95c6338f6\" (UID: \"626aa754-77ce-4648-8fa0-5ff95c6338f6\") "
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.336404    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.336402    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.336748 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.336426    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.337683 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.337056    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:22:03.337683 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.337141    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.337683 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.337239    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.337850 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.337697    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.337982 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.337954    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.338081 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.338052    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.338785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.338734    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:22:03.338904 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.338857    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:22:03.339160 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.339134    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.339365 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.339328    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.339540 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.339490    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-kube-api-access-b74kw" (OuterVolumeSpecName: "kube-api-access-b74kw") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "kube-api-access-b74kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:22:03.339651 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.339632    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-config" (OuterVolumeSpecName: "config") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.340036 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.340012    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-config-out" (OuterVolumeSpecName: "config-out") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:22:03.340125 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.340089    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.340221 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.340205    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.340417 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.340403    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.349904 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.349885    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-web-config" (OuterVolumeSpecName: "web-config") pod "626aa754-77ce-4648-8fa0-5ff95c6338f6" (UID: "626aa754-77ce-4648-8fa0-5ff95c6338f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:22:03.437624 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437598    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437624 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437623    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437639    2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-config-out\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437650    2553 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-config\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437664    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437677    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437690    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437705    2553 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-grpc-tls\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437717    2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-tls-assets\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437730    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437745    2553 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-kube-rbac-proxy\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437758    2553 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-metrics-client-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437773    2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-web-config\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437787    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437799    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/626aa754-77ce-4648-8fa0-5ff95c6338f6-prometheus-k8s-db\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437812    2553 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/626aa754-77ce-4648-8fa0-5ff95c6338f6-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437825    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74kw\" (UniqueName: \"kubernetes.io/projected/626aa754-77ce-4648-8fa0-5ff95c6338f6-kube-api-access-b74kw\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.437999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.437840    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/626aa754-77ce-4648-8fa0-5ff95c6338f6-configmap-metrics-client-ca\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:22:03.444076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444049    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc" exitCode=0
Sep 05 02:22:03.444076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444075    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8" exitCode=0
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444083    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0" exitCode=0
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444089    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f" exitCode=0
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444096    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a" exitCode=0
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444105    2553 generic.go:356] "Generic (PLEG): container finished" podID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50" exitCode=0
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444115    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"}
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"}
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444123    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.444247 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444236    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"}
Sep 05 02:22:03.444622 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"}
Sep 05 02:22:03.444622 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444285    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.444622 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444321    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"}
Sep 05 02:22:03.444622 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444363    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"}
Sep 05 02:22:03.444622 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.444406    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"626aa754-77ce-4648-8fa0-5ff95c6338f6","Type":"ContainerDied","Data":"3d2d3bef2892a3302e05144e8117a4a14a675f9ec9fe28f5cdece65d285c0768"}
Sep 05 02:22:03.452938 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.452924    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.459721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.459702    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.467088 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.467002    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.468480 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.468460    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:22:03.473096 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.473049    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:22:03.474385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.474367    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.480886 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.480869    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.487710 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.487694    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.494331 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.494315    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.494598 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.494580    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.494638 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.494606    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"} err="failed to get container status \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist"
Sep 05 02:22:03.494638 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.494623    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.494878 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.494861    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.494917 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.494882    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"} err="failed to get container status \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist"
Sep 05 02:22:03.494917 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.494895    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.495107 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.495090    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.495146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495110    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"} err="failed to get container status \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist"
Sep 05 02:22:03.495146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495122    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.495312 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.495297    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.495349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495315    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"} err="failed to get container status \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist"
Sep 05 02:22:03.495349 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495327    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.495514 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.495500    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.495548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495517    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"} err="failed to get container status \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist"
Sep 05 02:22:03.495548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495528    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.495758 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.495743    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.495802 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495762    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"} err="failed to get container status \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist"
Sep 05 02:22:03.495802 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495773    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.495991 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:22:03.495977    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.496028 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.495994    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"} err="failed to get container status \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist"
Sep 05 02:22:03.496028 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496005    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.496213 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496198    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"} err="failed to get container status \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist"
Sep 05 02:22:03.496253 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496213    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.496457 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496434    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"} err="failed to get container status \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist"
Sep 05 02:22:03.496523 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496458    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.496735 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496714    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"} err="failed to get container status \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist"
Sep 05 02:22:03.496735 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496733    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.496948 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496932    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"} err="failed to get container status \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist"
Sep 05 02:22:03.496990 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.496950    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.497169 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497154    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"} err="failed to get container status \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist"
Sep 05 02:22:03.497209 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497170    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.497372 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497357    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"} err="failed to get container status \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist"
Sep 05 02:22:03.497413 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497373    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.497624 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497606    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"} err="failed to get container status \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist"
Sep 05 02:22:03.497662 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497626    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.497841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497826    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"} err="failed to get container status \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist"
Sep 05 02:22:03.497875 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.497842    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.498061 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498039    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"} err="failed to get container status \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist"
Sep 05 02:22:03.498061 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498060    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.498276 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498261    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"} err="failed to get container status \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist"
Sep 05 02:22:03.498325 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498276    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.498483 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498465    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"} err="failed to get container status \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist"
Sep 05 02:22:03.498553 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498486    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.498719 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498704    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"} err="failed to get container status \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist"
Sep 05 02:22:03.498774 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498719    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.498925 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498909    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"} err="failed to get container status \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist"
Sep 05 02:22:03.498970 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.498925    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.499104 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.499088    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"} err="failed to get container status \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist"
Sep 05 02:22:03.499145 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.499104    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.499307 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.499290    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"} err="failed to get container status \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist"
Sep 05 02:22:03.499370 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.499309    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.499758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.499721    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"} err="failed to get container status \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist"
Sep 05 02:22:03.499863 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.499758    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.500398 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.500375    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"} err="failed to get container status \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist"
Sep 05 02:22:03.500398 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.500398    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.500737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.500707    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"} err="failed to get container status \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist"
Sep 05 02:22:03.500807 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.500739    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.501020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501001    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"} err="failed to get container status \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist"
Sep 05 02:22:03.501067 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501022    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.501263 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501245    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"} err="failed to get container status \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist"
Sep 05 02:22:03.501340 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501265    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.501503 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501482    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"} err="failed to get container status \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist"
Sep 05 02:22:03.501548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501506    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.501600 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501553    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:22:03.501795 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501777    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"} err="failed to get container status \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist"
Sep 05 02:22:03.501870 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501796    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.501920 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501906    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-web"
Sep 05 02:22:03.501957 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501925    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-web"
Sep 05 02:22:03.501957 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501940    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="config-reloader"
Sep 05 02:22:03.501957 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501949    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="config-reloader"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501960    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501968    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501982    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="prometheus"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.501991    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="prometheus"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502004    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="thanos-sidecar"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502014    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="thanos-sidecar"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502026    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-thanos"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502031    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-thanos"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502043    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="init-config-reloader"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502042    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"} err="failed to get container status \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist"
Sep 05 02:22:03.502059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502059    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502053    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="init-config-reloader"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502177    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="config-reloader"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502193    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="thanos-sidecar"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502203    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502214    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-thanos"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502223    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="kube-rbac-proxy-web"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502229    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" containerName="prometheus"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502288    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"} err="failed to get container status \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist"
Sep 05 02:22:03.502444 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502314    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.502764 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502552    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"} err="failed to get container status \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist"
Sep 05 02:22:03.502764 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502585    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.502982 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502954    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"} err="failed to get container status \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist"
Sep 05 02:22:03.503051 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.502984    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.503311 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.503278    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"} err="failed to get container status \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist"
Sep 05 02:22:03.503311 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.503313    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.503620 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.503599    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"} err="failed to get container status \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist"
Sep 05 02:22:03.503705 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.503621    2553 scope.go:117] "RemoveContainer" containerID="f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"
Sep 05 02:22:03.503904 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.503884    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc"} err="failed to get container status \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": rpc error: code = NotFound desc = could not find container \"f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc\": container with ID starting with f82d918cc26f4610a5524194f7862730fe357e3a414746841c0de28bf431ebfc not found: ID does not exist"
Sep 05 02:22:03.503945 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.503905    2553 scope.go:117] "RemoveContainer" containerID="69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"
Sep 05 02:22:03.504148 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504130    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8"} err="failed to get container status \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": rpc error: code = NotFound desc = could not find container \"69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8\": container with ID starting with 69b3ff12cc059d041dd61221eb48396431d0c8a07397ad819a060008dcd290c8 not found: ID does not exist"
Sep 05 02:22:03.504228 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504149    2553 scope.go:117] "RemoveContainer" containerID="9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"
Sep 05 02:22:03.504403 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504381    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0"} err="failed to get container status \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": rpc error: code = NotFound desc = could not find container \"9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0\": container with ID starting with 9cdc830789c8f6039cede75d9e8f7fbc292966f72ba20ba6c57c1619b9d1bbb0 not found: ID does not exist"
Sep 05 02:22:03.504403 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504403    2553 scope.go:117] "RemoveContainer" containerID="c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"
Sep 05 02:22:03.504664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504647    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f"} err="failed to get container status \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": rpc error: code = NotFound desc = could not find container \"c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f\": container with ID starting with c9dbe35e3ea50c1438fdab330844552cdd79e052205c892e7f723b71fc78b20f not found: ID does not exist"
Sep 05 02:22:03.504664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504664    2553 scope.go:117] "RemoveContainer" containerID="9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"
Sep 05 02:22:03.504899 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504880    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a"} err="failed to get container status \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": rpc error: code = NotFound desc = could not find container \"9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a\": container with ID starting with 9bf02e236ab67204be8c9d29213a48cd90b444f254da489868cc8ad59d15880a not found: ID does not exist"
Sep 05 02:22:03.504954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.504900    2553 scope.go:117] "RemoveContainer" containerID="eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"
Sep 05 02:22:03.505143 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.505124    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50"} err="failed to get container status \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": rpc error: code = NotFound desc = could not find container \"eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50\": container with ID starting with eb1e09c19f42e9e3d03ac74ebc108cc1c28ea5795ec8f6771867ace065be1d50 not found: ID does not exist"
Sep 05 02:22:03.505193 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.505145    2553 scope.go:117] "RemoveContainer" containerID="bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"
Sep 05 02:22:03.505409 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.505393    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830"} err="failed to get container status \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": rpc error: code = NotFound desc = could not find container \"bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830\": container with ID starting with bfd70ebb05a7a35b68b59069ed32a1c850ea45e36e24f0d885d8f0b9aac3d830 not found: ID does not exist"
Sep 05 02:22:03.507946 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.507929    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.510863 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.510840    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 05 02:22:03.510972 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.510843    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 05 02:22:03.510972 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.510884    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 05 02:22:03.510972 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.510843    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511233    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511247    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511261    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2sbts\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511284    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511308    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511322    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 05 02:22:03.511371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511286    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3adg565164sbp\""
Sep 05 02:22:03.511736 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511421    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 05 02:22:03.511736 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511729    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 05 02:22:03.511980 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.511966    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 05 02:22:03.513978 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.513961    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 05 02:22:03.516695 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.516675    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 05 02:22:03.522059 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.522040    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:22:03.639075 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639046    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639075 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639073    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639238 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639088    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639238 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639107    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639238 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639165    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639238 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639207    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639238 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639231    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639298    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639325    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639347    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639371    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-config\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639385 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639384    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4634099a-9a65-41d2-992e-f6651990d75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639629 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639412    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639629 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639457    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639629 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639519    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639629 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639590    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639629 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639617    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n224w\" (UniqueName: \"kubernetes.io/projected/4634099a-9a65-41d2-992e-f6651990d75a-kube-api-access-n224w\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639831 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639645    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4634099a-9a65-41d2-992e-f6651990d75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639831 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639677    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.639831 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.639750    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741187 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741106    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741178    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741268    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741292    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741329    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741360    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741399    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741520    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741542    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741605    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-config\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.741627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741630    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4634099a-9a65-41d2-992e-f6651990d75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741679    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741712    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741740    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741766    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n224w\" (UniqueName: \"kubernetes.io/projected/4634099a-9a65-41d2-992e-f6651990d75a-kube-api-access-n224w\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741797    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4634099a-9a65-41d2-992e-f6651990d75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741903    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741927    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.742020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.741936    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.743311 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.742787    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.743311 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.743111    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.743534 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.743375    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.743834 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.743809    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.744080 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.744057    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.744532 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.744481    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4634099a-9a65-41d2-992e-f6651990d75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.744682 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.744657    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-config\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.744766 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.744745    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.745665 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.745641    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4634099a-9a65-41d2-992e-f6651990d75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.745814 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.745746    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4634099a-9a65-41d2-992e-f6651990d75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.746336 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.746309    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.746602 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.746585    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.746689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.746647    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.747061 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.747032    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.747159 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.747135    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.747314 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.747295    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4634099a-9a65-41d2-992e-f6651990d75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.751242 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.751221    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n224w\" (UniqueName: \"kubernetes.io/projected/4634099a-9a65-41d2-992e-f6651990d75a-kube-api-access-n224w\") pod \"prometheus-k8s-0\" (UID: \"4634099a-9a65-41d2-992e-f6651990d75a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:03.818811 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:03.818790    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:22:04.098859 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:04.098825    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 05 02:22:04.100765 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:22:04.100740    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4634099a_9a65_41d2_992e_f6651990d75a.slice/crio-82384a7da3d46186cac18bb2d418bbf63f9f0bae24a7a6f563b1680d00f72a28 WatchSource:0}: Error finding container 82384a7da3d46186cac18bb2d418bbf63f9f0bae24a7a6f563b1680d00f72a28: Status 404 returned error can't find the container with id 82384a7da3d46186cac18bb2d418bbf63f9f0bae24a7a6f563b1680d00f72a28
Sep 05 02:22:04.392131 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:04.392095    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626aa754-77ce-4648-8fa0-5ff95c6338f6" path="/var/lib/kubelet/pods/626aa754-77ce-4648-8fa0-5ff95c6338f6/volumes"
Sep 05 02:22:04.450621 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:04.450590    2553 generic.go:356] "Generic (PLEG): container finished" podID="4634099a-9a65-41d2-992e-f6651990d75a" containerID="7e6f6a5c605a7147b77c5545e5808a4409ec2ebecd2c265d6ba4a0414d790773" exitCode=0
Sep 05 02:22:04.451006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:04.450670    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerDied","Data":"7e6f6a5c605a7147b77c5545e5808a4409ec2ebecd2c265d6ba4a0414d790773"}
Sep 05 02:22:04.451006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:04.450733    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"82384a7da3d46186cac18bb2d418bbf63f9f0bae24a7a6f563b1680d00f72a28"}
Sep 05 02:22:05.459550 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.459512    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"c54c9afb7430f69d0a4021455ebfdc0f712752f8ee8eeee20209ac7ca43ca862"}
Sep 05 02:22:05.460016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.459608    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"fee8d3fc1b85b21e69712f1c64e9de10375c93ab97dd765a836821dab9d6e4b0"}
Sep 05 02:22:05.460016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.459651    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"a2d0826e9dad3f02fe4c67de7f58c202168b01995449b3b541b5ad336c1c89fb"}
Sep 05 02:22:05.460016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.459695    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"08894dd69ccff8710cf53ff07791f960327b5d6ffdb5881e121f8e407f4289da"}
Sep 05 02:22:05.460016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.459736    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"b0c4427eed719440bbe5ac61c50b075d7e70fae3578e7be954d842ef211f1a68"}
Sep 05 02:22:05.460016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.459772    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4634099a-9a65-41d2-992e-f6651990d75a","Type":"ContainerStarted","Data":"6c217f6bf9763caf88953ced51780859cad1b45b46c39dc68a648941bd9171f0"}
Sep 05 02:22:05.493332 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:05.493277    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.474915127 podStartE2EDuration="2.493257112s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.341985ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:22:05.493196391 +0000 UTC m=+235.692559398" watchObservedRunningTime="2026-09-05 02:22:05.493257112 +0000 UTC m=+235.692620105"
Sep 05 02:22:08.819723 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:22:08.819682    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:23:03.819704 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:23:03.819656    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:23:03.835271 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:23:03.835236    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:23:04.724655 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:23:04.724625    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 05 02:23:10.189859 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:23:10.189827    2553 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:25:57.282492 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.282452    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bjl47"]
Sep 05 02:25:57.286951 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.286914    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.291264 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.291236    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 05 02:25:57.291264 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.291236    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nzs4d\""
Sep 05 02:25:57.292004 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.291986    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 05 02:25:57.298997 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.298968    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bjl47"]
Sep 05 02:25:57.409449 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.409402    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r54\" (UniqueName: \"kubernetes.io/projected/46cb9c06-5edd-4a3e-8239-08c95453587c-kube-api-access-64r54\") pod \"cert-manager-webhook-6486ffd997-bjl47\" (UID: \"46cb9c06-5edd-4a3e-8239-08c95453587c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.409651 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.409465    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46cb9c06-5edd-4a3e-8239-08c95453587c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bjl47\" (UID: \"46cb9c06-5edd-4a3e-8239-08c95453587c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.510223 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.510187    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r54\" (UniqueName: \"kubernetes.io/projected/46cb9c06-5edd-4a3e-8239-08c95453587c-kube-api-access-64r54\") pod \"cert-manager-webhook-6486ffd997-bjl47\" (UID: \"46cb9c06-5edd-4a3e-8239-08c95453587c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.510408 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.510259    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46cb9c06-5edd-4a3e-8239-08c95453587c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bjl47\" (UID: \"46cb9c06-5edd-4a3e-8239-08c95453587c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.524509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.524470    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46cb9c06-5edd-4a3e-8239-08c95453587c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bjl47\" (UID: \"46cb9c06-5edd-4a3e-8239-08c95453587c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.524509 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.524501    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r54\" (UniqueName: \"kubernetes.io/projected/46cb9c06-5edd-4a3e-8239-08c95453587c-kube-api-access-64r54\") pod \"cert-manager-webhook-6486ffd997-bjl47\" (UID: \"46cb9c06-5edd-4a3e-8239-08c95453587c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.611125 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.611031    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:25:57.897605 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.897489    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bjl47"]
Sep 05 02:25:57.901232 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:25:57.901202    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46cb9c06_5edd_4a3e_8239_08c95453587c.slice/crio-9b4a55adefa1b8fd273b85eea3d2787c76e094efe19bbc6bd3fd1ddd694b9782 WatchSource:0}: Error finding container 9b4a55adefa1b8fd273b85eea3d2787c76e094efe19bbc6bd3fd1ddd694b9782: Status 404 returned error can't find the container with id 9b4a55adefa1b8fd273b85eea3d2787c76e094efe19bbc6bd3fd1ddd694b9782
Sep 05 02:25:57.903312 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:57.903289    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:25:58.397103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:25:58.397069    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47" event={"ID":"46cb9c06-5edd-4a3e-8239-08c95453587c","Type":"ContainerStarted","Data":"9b4a55adefa1b8fd273b85eea3d2787c76e094efe19bbc6bd3fd1ddd694b9782"}
Sep 05 02:26:01.412465 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:01.412435    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47" event={"ID":"46cb9c06-5edd-4a3e-8239-08c95453587c","Type":"ContainerStarted","Data":"8e1e0250497e82365b6f268cb0eb6293d20ac680bdd1443c381b96e1daf632c1"}
Sep 05 02:26:01.412872 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:01.412518    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:26:07.418720 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:07.418678    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47"
Sep 05 02:26:07.439685 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:07.439625    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bjl47" podStartSLOduration=7.580756944 podStartE2EDuration="10.439609726s" totalImagesPullingTime="2.858852782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:01.451658422 +0000 UTC m=+471.651021419" watchObservedRunningTime="2026-09-05 02:26:07.439609726 +0000 UTC m=+477.638972719"
Sep 05 02:26:09.487145 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.487111    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"]
Sep 05 02:26:09.490570 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.490544    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.493426 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.493402    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:26:09.493426 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.493412    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 05 02:26:09.493618 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.493462    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fv22n\""
Sep 05 02:26:09.500194 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.500170    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"]
Sep 05 02:26:09.618323 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.618285    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc9c28ec-6277-4a55-95ac-6b80cd9208a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wqc59\" (UID: \"bc9c28ec-6277-4a55-95ac-6b80cd9208a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.618523 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.618418    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnb8c\" (UniqueName: \"kubernetes.io/projected/bc9c28ec-6277-4a55-95ac-6b80cd9208a8-kube-api-access-vnb8c\") pod \"openshift-lws-operator-fd8ccff4c-wqc59\" (UID: \"bc9c28ec-6277-4a55-95ac-6b80cd9208a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.719272 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.719238    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnb8c\" (UniqueName: \"kubernetes.io/projected/bc9c28ec-6277-4a55-95ac-6b80cd9208a8-kube-api-access-vnb8c\") pod \"openshift-lws-operator-fd8ccff4c-wqc59\" (UID: \"bc9c28ec-6277-4a55-95ac-6b80cd9208a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.719431 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.719293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc9c28ec-6277-4a55-95ac-6b80cd9208a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wqc59\" (UID: \"bc9c28ec-6277-4a55-95ac-6b80cd9208a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.719737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.719719    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc9c28ec-6277-4a55-95ac-6b80cd9208a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wqc59\" (UID: \"bc9c28ec-6277-4a55-95ac-6b80cd9208a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.727482 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.727455    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnb8c\" (UniqueName: \"kubernetes.io/projected/bc9c28ec-6277-4a55-95ac-6b80cd9208a8-kube-api-access-vnb8c\") pod \"openshift-lws-operator-fd8ccff4c-wqc59\" (UID: \"bc9c28ec-6277-4a55-95ac-6b80cd9208a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:09.800326 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:09.800292    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"
Sep 05 02:26:10.075258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:10.075231    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59"]
Sep 05 02:26:10.077612 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:26:10.077579    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc9c28ec_6277_4a55_95ac_6b80cd9208a8.slice/crio-c30ce9ea9c0352e2c2f2b023af0bf52f1bfc68f1eab9d28c1adea8e6b0bd0630 WatchSource:0}: Error finding container c30ce9ea9c0352e2c2f2b023af0bf52f1bfc68f1eab9d28c1adea8e6b0bd0630: Status 404 returned error can't find the container with id c30ce9ea9c0352e2c2f2b023af0bf52f1bfc68f1eab9d28c1adea8e6b0bd0630
Sep 05 02:26:10.449923 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:10.449837    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59" event={"ID":"bc9c28ec-6277-4a55-95ac-6b80cd9208a8","Type":"ContainerStarted","Data":"c30ce9ea9c0352e2c2f2b023af0bf52f1bfc68f1eab9d28c1adea8e6b0bd0630"}
Sep 05 02:26:13.463861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:13.463825    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59" event={"ID":"bc9c28ec-6277-4a55-95ac-6b80cd9208a8","Type":"ContainerStarted","Data":"ec7ff7db8df53ee7ab1780ac2962e4b363693d792f25ced9c65fa088e65e7f90"}
Sep 05 02:26:13.482071 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:13.482010    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wqc59" podStartSLOduration=1.604076616 podStartE2EDuration="4.481988371s" totalImagesPullingTime="2.877911755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:13.48029639 +0000 UTC m=+483.679659386" watchObservedRunningTime="2026-09-05 02:26:13.481988371 +0000 UTC m=+483.681351370"
Sep 05 02:26:31.916145 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.916109    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"]
Sep 05 02:26:31.922766 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.922742    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:31.927850 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.927818    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wwndb\""
Sep 05 02:26:31.928005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.927853    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 05 02:26:31.928173 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.928148    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 05 02:26:31.928693 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.928675    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 05 02:26:31.943294 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:31.943264    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"]
Sep 05 02:26:32.013792 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.013752    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-cert\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.013968 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.013825    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-metrics-cert\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.013968 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.013870    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-manager-config\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.013968 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.013914    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wbf\" (UniqueName: \"kubernetes.io/projected/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-kube-api-access-g6wbf\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.114553 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.114511    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-manager-config\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.114794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.114609    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wbf\" (UniqueName: \"kubernetes.io/projected/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-kube-api-access-g6wbf\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.114794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.114670    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-cert\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.114794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.114735    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-metrics-cert\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.115364 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.115337    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-manager-config\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.117193 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.117170    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-metrics-cert\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.117294 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.117235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-cert\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.123097 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.123065    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wbf\" (UniqueName: \"kubernetes.io/projected/ee9b3946-f9a1-48b3-bab6-dfb23af859c8-kube-api-access-g6wbf\") pod \"lws-controller-manager-54b595bc86-765tw\" (UID: \"ee9b3946-f9a1-48b3-bab6-dfb23af859c8\") " pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.232167 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.232044    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:32.521654 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.519672    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"]
Sep 05 02:26:32.546499 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:32.546467    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw" event={"ID":"ee9b3946-f9a1-48b3-bab6-dfb23af859c8","Type":"ContainerStarted","Data":"61dd241a2825a8a3a8305f2f4411c59da8564c299eba4c5f5b807e22cbbb8179"}
Sep 05 02:26:35.561389 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:35.561350    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw" event={"ID":"ee9b3946-f9a1-48b3-bab6-dfb23af859c8","Type":"ContainerStarted","Data":"b7106f721c47e0760f0c55e0d7be1a0cc0f160881158b0401559884a9419ebeb"}
Sep 05 02:26:35.561903 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:35.561439    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:26:35.578836 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:35.578727    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw" podStartSLOduration=1.7787056140000002 podStartE2EDuration="4.578707925s" totalImagesPullingTime="2.800002311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:35.578316278 +0000 UTC m=+505.777679284" watchObservedRunningTime="2026-09-05 02:26:35.578707925 +0000 UTC m=+505.778070897"
Sep 05 02:26:46.568192 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:26:46.568158    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54b595bc86-765tw"
Sep 05 02:27:00.838804 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.838769    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"]
Sep 05 02:27:00.841365 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.841347    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:00.844181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.844140    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 05 02:27:00.844337 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.844260    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zhnpc\""
Sep 05 02:27:00.844453 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.844434    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 05 02:27:00.844556 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.844538    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 05 02:27:00.866968 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.866926    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"]
Sep 05 02:27:00.973452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:00.973409    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9b7t\" (UniqueName: \"kubernetes.io/projected/ce9fdeba-e8a3-4b60-92de-a779b11f760e-kube-api-access-f9b7t\") pod \"dns-operator-controller-manager-ccbdf686-msqjr\" (UID: \"ce9fdeba-e8a3-4b60-92de-a779b11f760e\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:01.074487 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:01.074445    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9b7t\" (UniqueName: \"kubernetes.io/projected/ce9fdeba-e8a3-4b60-92de-a779b11f760e-kube-api-access-f9b7t\") pod \"dns-operator-controller-manager-ccbdf686-msqjr\" (UID: \"ce9fdeba-e8a3-4b60-92de-a779b11f760e\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:01.086615 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:01.086579    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9b7t\" (UniqueName: \"kubernetes.io/projected/ce9fdeba-e8a3-4b60-92de-a779b11f760e-kube-api-access-f9b7t\") pod \"dns-operator-controller-manager-ccbdf686-msqjr\" (UID: \"ce9fdeba-e8a3-4b60-92de-a779b11f760e\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:01.152368 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:01.152272    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:01.433600 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:01.433543    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"]
Sep 05 02:27:01.436724 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:27:01.436686    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9fdeba_e8a3_4b60_92de_a779b11f760e.slice/crio-54317e856cdd52e06bf8524124c76f5977dae4759718e48604de9a3f610b7122 WatchSource:0}: Error finding container 54317e856cdd52e06bf8524124c76f5977dae4759718e48604de9a3f610b7122: Status 404 returned error can't find the container with id 54317e856cdd52e06bf8524124c76f5977dae4759718e48604de9a3f610b7122
Sep 05 02:27:01.674341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:01.674303    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr" event={"ID":"ce9fdeba-e8a3-4b60-92de-a779b11f760e","Type":"ContainerStarted","Data":"54317e856cdd52e06bf8524124c76f5977dae4759718e48604de9a3f610b7122"}
Sep 05 02:27:04.694376 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:04.694282    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr" event={"ID":"ce9fdeba-e8a3-4b60-92de-a779b11f760e","Type":"ContainerStarted","Data":"48bee8a1cc16d188e3d8ee05a4ebbb327b46504fa088bdf9e633666fcc1d661b"}
Sep 05 02:27:04.694376 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:04.694371    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:04.739789 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:04.739732    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr" podStartSLOduration=1.851791349 podStartE2EDuration="4.739711162s" totalImagesPullingTime="2.887919813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:04.737951865 +0000 UTC m=+534.937314860" watchObservedRunningTime="2026-09-05 02:27:04.739711162 +0000 UTC m=+534.939074155"
Sep 05 02:27:05.714089 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.714053    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-4d9fp"]
Sep 05 02:27:05.718534 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.718510    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:05.725464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.725441    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x82db\""
Sep 05 02:27:05.733950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.733915    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-4d9fp"]
Sep 05 02:27:05.818323 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.818285    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7qm\" (UniqueName: \"kubernetes.io/projected/5084691f-d6e5-4464-b80c-45e447756f69-kube-api-access-ll7qm\") pod \"authorino-operator-56fb6f6856-4d9fp\" (UID: \"5084691f-d6e5-4464-b80c-45e447756f69\") " pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:05.919355 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.919306    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7qm\" (UniqueName: \"kubernetes.io/projected/5084691f-d6e5-4464-b80c-45e447756f69-kube-api-access-ll7qm\") pod \"authorino-operator-56fb6f6856-4d9fp\" (UID: \"5084691f-d6e5-4464-b80c-45e447756f69\") " pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:05.932375 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:05.932346    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7qm\" (UniqueName: \"kubernetes.io/projected/5084691f-d6e5-4464-b80c-45e447756f69-kube-api-access-ll7qm\") pod \"authorino-operator-56fb6f6856-4d9fp\" (UID: \"5084691f-d6e5-4464-b80c-45e447756f69\") " pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:06.030180 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:06.030153    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:06.306198 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:06.306135    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-4d9fp"]
Sep 05 02:27:06.308637 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:27:06.308599    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5084691f_d6e5_4464_b80c_45e447756f69.slice/crio-329847e6b98cbf0f6020bf9d4c3c9b821b5953a5a3fbeaa97484dc474ee3e592 WatchSource:0}: Error finding container 329847e6b98cbf0f6020bf9d4c3c9b821b5953a5a3fbeaa97484dc474ee3e592: Status 404 returned error can't find the container with id 329847e6b98cbf0f6020bf9d4c3c9b821b5953a5a3fbeaa97484dc474ee3e592
Sep 05 02:27:06.707414 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:06.707316    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp" event={"ID":"5084691f-d6e5-4464-b80c-45e447756f69","Type":"ContainerStarted","Data":"329847e6b98cbf0f6020bf9d4c3c9b821b5953a5a3fbeaa97484dc474ee3e592"}
Sep 05 02:27:08.718306 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:08.718268    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp" event={"ID":"5084691f-d6e5-4464-b80c-45e447756f69","Type":"ContainerStarted","Data":"17ac1bd895b329616e1e3ca53ffde3d9662e5eae007b1da9b8eef99c9516271a"}
Sep 05 02:27:08.718783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:08.718331    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:08.736657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:08.736601    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp" podStartSLOduration=1.793707275 podStartE2EDuration="3.736583463s" totalImagesPullingTime="1.942876188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:08.735552466 +0000 UTC m=+538.934915461" watchObservedRunningTime="2026-09-05 02:27:08.736583463 +0000 UTC m=+538.935946509"
Sep 05 02:27:15.703894 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:15.703861    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:27:19.725964 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:19.725935    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-4d9fp"
Sep 05 02:27:51.396661 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.396628    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-nzvzg"]
Sep 05 02:27:51.401852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.401831    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:51.404448 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.404425    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tsfvk\""
Sep 05 02:27:51.405148 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.405129    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nzvzg"]
Sep 05 02:27:51.506931 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.506903    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvntx\" (UniqueName: \"kubernetes.io/projected/267a185e-95ea-492c-a6a1-2d4dc22bab06-kube-api-access-qvntx\") pod \"authorino-b54df94b4-nzvzg\" (UID: \"267a185e-95ea-492c-a6a1-2d4dc22bab06\") " pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:51.554649 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.554623    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8t6wq"]
Sep 05 02:27:51.558041 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.558023    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:27:51.568730 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.568708    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8t6wq"]
Sep 05 02:27:51.608357 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.608326    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvntx\" (UniqueName: \"kubernetes.io/projected/267a185e-95ea-492c-a6a1-2d4dc22bab06-kube-api-access-qvntx\") pod \"authorino-b54df94b4-nzvzg\" (UID: \"267a185e-95ea-492c-a6a1-2d4dc22bab06\") " pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:51.616894 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.616871    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvntx\" (UniqueName: \"kubernetes.io/projected/267a185e-95ea-492c-a6a1-2d4dc22bab06-kube-api-access-qvntx\") pod \"authorino-b54df94b4-nzvzg\" (UID: \"267a185e-95ea-492c-a6a1-2d4dc22bab06\") " pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:51.708785 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.708714    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zw8\" (UniqueName: \"kubernetes.io/projected/49f5861c-3b4a-4690-b935-b5f93ba52177-kube-api-access-l5zw8\") pod \"authorino-c548b5d7d-8t6wq\" (UID: \"49f5861c-3b4a-4690-b935-b5f93ba52177\") " pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:27:51.712665 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.712647    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:51.810125 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.810091    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zw8\" (UniqueName: \"kubernetes.io/projected/49f5861c-3b4a-4690-b935-b5f93ba52177-kube-api-access-l5zw8\") pod \"authorino-c548b5d7d-8t6wq\" (UID: \"49f5861c-3b4a-4690-b935-b5f93ba52177\") " pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:27:51.818586 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.818545    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zw8\" (UniqueName: \"kubernetes.io/projected/49f5861c-3b4a-4690-b935-b5f93ba52177-kube-api-access-l5zw8\") pod \"authorino-c548b5d7d-8t6wq\" (UID: \"49f5861c-3b4a-4690-b935-b5f93ba52177\") " pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:27:51.866997 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.866971    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:27:51.986886 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:51.986837    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nzvzg"]
Sep 05 02:27:51.989165 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:27:51.989139    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod267a185e_95ea_492c_a6a1_2d4dc22bab06.slice/crio-369d89d5c21a22f52d68106a02f0d8dec53c6c4a9ee28fed1075f605099a557a WatchSource:0}: Error finding container 369d89d5c21a22f52d68106a02f0d8dec53c6c4a9ee28fed1075f605099a557a: Status 404 returned error can't find the container with id 369d89d5c21a22f52d68106a02f0d8dec53c6c4a9ee28fed1075f605099a557a
Sep 05 02:27:52.133954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:52.133928    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8t6wq"]
Sep 05 02:27:52.135576 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:27:52.135537    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f5861c_3b4a_4690_b935_b5f93ba52177.slice/crio-e4a25274b41df95efb881f99b2e5c2abac1429dfc0ab0f4a9367098f53c1533e WatchSource:0}: Error finding container e4a25274b41df95efb881f99b2e5c2abac1429dfc0ab0f4a9367098f53c1533e: Status 404 returned error can't find the container with id e4a25274b41df95efb881f99b2e5c2abac1429dfc0ab0f4a9367098f53c1533e
Sep 05 02:27:52.925513 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:52.925473    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8t6wq" event={"ID":"49f5861c-3b4a-4690-b935-b5f93ba52177","Type":"ContainerStarted","Data":"e4a25274b41df95efb881f99b2e5c2abac1429dfc0ab0f4a9367098f53c1533e"}
Sep 05 02:27:52.927212 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:52.927181    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nzvzg" event={"ID":"267a185e-95ea-492c-a6a1-2d4dc22bab06","Type":"ContainerStarted","Data":"369d89d5c21a22f52d68106a02f0d8dec53c6c4a9ee28fed1075f605099a557a"}
Sep 05 02:27:54.939961 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:54.939911    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nzvzg" event={"ID":"267a185e-95ea-492c-a6a1-2d4dc22bab06","Type":"ContainerStarted","Data":"ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550"}
Sep 05 02:27:54.941484 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:54.941456    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8t6wq" event={"ID":"49f5861c-3b4a-4690-b935-b5f93ba52177","Type":"ContainerStarted","Data":"17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309"}
Sep 05 02:27:54.957476 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:54.957436    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-nzvzg" podStartSLOduration=1.759921839 podStartE2EDuration="3.957424423s" totalImagesPullingTime="2.197502584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:54.956090341 +0000 UTC m=+585.155453335" watchObservedRunningTime="2026-09-05 02:27:54.957424423 +0000 UTC m=+585.156787432"
Sep 05 02:27:54.975380 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:54.975319    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-8t6wq" podStartSLOduration=1.934596715 podStartE2EDuration="3.975300914s" totalImagesPullingTime="2.040704199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:54.973986154 +0000 UTC m=+585.173349148" watchObservedRunningTime="2026-09-05 02:27:54.975300914 +0000 UTC m=+585.174663909"
Sep 05 02:27:55.003101 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:55.003071    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nzvzg"]
Sep 05 02:27:56.950668 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:56.950603    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-nzvzg" podUID="267a185e-95ea-492c-a6a1-2d4dc22bab06" containerName="authorino" containerID="cri-o://ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550" gracePeriod=30
Sep 05 02:27:57.191135 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.191114    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:57.359112 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.359084    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvntx\" (UniqueName: \"kubernetes.io/projected/267a185e-95ea-492c-a6a1-2d4dc22bab06-kube-api-access-qvntx\") pod \"267a185e-95ea-492c-a6a1-2d4dc22bab06\" (UID: \"267a185e-95ea-492c-a6a1-2d4dc22bab06\") "
Sep 05 02:27:57.361190 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.361163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267a185e-95ea-492c-a6a1-2d4dc22bab06-kube-api-access-qvntx" (OuterVolumeSpecName: "kube-api-access-qvntx") pod "267a185e-95ea-492c-a6a1-2d4dc22bab06" (UID: "267a185e-95ea-492c-a6a1-2d4dc22bab06"). InnerVolumeSpecName "kube-api-access-qvntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:27:57.460523 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.460491    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvntx\" (UniqueName: \"kubernetes.io/projected/267a185e-95ea-492c-a6a1-2d4dc22bab06-kube-api-access-qvntx\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:27:57.956551 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.956516    2553 generic.go:356] "Generic (PLEG): container finished" podID="267a185e-95ea-492c-a6a1-2d4dc22bab06" containerID="ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550" exitCode=0
Sep 05 02:27:57.957006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.956569    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nzvzg"
Sep 05 02:27:57.957006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.956584    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nzvzg" event={"ID":"267a185e-95ea-492c-a6a1-2d4dc22bab06","Type":"ContainerDied","Data":"ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550"}
Sep 05 02:27:57.957006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.956616    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nzvzg" event={"ID":"267a185e-95ea-492c-a6a1-2d4dc22bab06","Type":"ContainerDied","Data":"369d89d5c21a22f52d68106a02f0d8dec53c6c4a9ee28fed1075f605099a557a"}
Sep 05 02:27:57.957006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.956632    2553 scope.go:117] "RemoveContainer" containerID="ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550"
Sep 05 02:27:57.967209 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.967189    2553 scope.go:117] "RemoveContainer" containerID="ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550"
Sep 05 02:27:57.967475 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:27:57.967455    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550\": container with ID starting with ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550 not found: ID does not exist" containerID="ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550"
Sep 05 02:27:57.967536 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.967483    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550"} err="failed to get container status \"ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550\": rpc error: code = NotFound desc = could not find container \"ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550\": container with ID starting with ebf25b8e74404ea331e18c2d29741cd26e6e540dc81fe842cd22ffc5ae03a550 not found: ID does not exist"
Sep 05 02:27:57.980733 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.980691    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nzvzg"]
Sep 05 02:27:57.982476 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:57.982456    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nzvzg"]
Sep 05 02:27:58.392713 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:27:58.392681    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="267a185e-95ea-492c-a6a1-2d4dc22bab06" path="/var/lib/kubelet/pods/267a185e-95ea-492c-a6a1-2d4dc22bab06/volumes"
Sep 05 02:28:14.306433 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.306397    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-c26gd"]
Sep 05 02:28:14.306861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.306767    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="267a185e-95ea-492c-a6a1-2d4dc22bab06" containerName="authorino"
Sep 05 02:28:14.306861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.306784    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="267a185e-95ea-492c-a6a1-2d4dc22bab06" containerName="authorino"
Sep 05 02:28:14.306975 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.306892    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="267a185e-95ea-492c-a6a1-2d4dc22bab06" containerName="authorino"
Sep 05 02:28:14.311239 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.311221    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.314273 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.314255    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 05 02:28:14.319798 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.319773    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-c26gd"]
Sep 05 02:28:14.404741 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.404703    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7pp\" (UniqueName: \"kubernetes.io/projected/4caf73f9-59bd-4e2f-ace3-39b25bc5e352-kube-api-access-9r7pp\") pod \"authorino-7cf4765b85-c26gd\" (UID: \"4caf73f9-59bd-4e2f-ace3-39b25bc5e352\") " pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.404913 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.404799    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4caf73f9-59bd-4e2f-ace3-39b25bc5e352-tls-cert\") pod \"authorino-7cf4765b85-c26gd\" (UID: \"4caf73f9-59bd-4e2f-ace3-39b25bc5e352\") " pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.506090 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.506044    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7pp\" (UniqueName: \"kubernetes.io/projected/4caf73f9-59bd-4e2f-ace3-39b25bc5e352-kube-api-access-9r7pp\") pod \"authorino-7cf4765b85-c26gd\" (UID: \"4caf73f9-59bd-4e2f-ace3-39b25bc5e352\") " pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.506241 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.506109    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4caf73f9-59bd-4e2f-ace3-39b25bc5e352-tls-cert\") pod \"authorino-7cf4765b85-c26gd\" (UID: \"4caf73f9-59bd-4e2f-ace3-39b25bc5e352\") " pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.508537 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.508519    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4caf73f9-59bd-4e2f-ace3-39b25bc5e352-tls-cert\") pod \"authorino-7cf4765b85-c26gd\" (UID: \"4caf73f9-59bd-4e2f-ace3-39b25bc5e352\") " pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.513971 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.513946    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7pp\" (UniqueName: \"kubernetes.io/projected/4caf73f9-59bd-4e2f-ace3-39b25bc5e352-kube-api-access-9r7pp\") pod \"authorino-7cf4765b85-c26gd\" (UID: \"4caf73f9-59bd-4e2f-ace3-39b25bc5e352\") " pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.620392 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.620301    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-c26gd"
Sep 05 02:28:14.890472 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:14.890445    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-c26gd"]
Sep 05 02:28:14.892834 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:28:14.892804    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4caf73f9_59bd_4e2f_ace3_39b25bc5e352.slice/crio-96ad68d03b2bb27af4b8d7309ab2b3262a2abd2f052600483f081f4ca835a6cd WatchSource:0}: Error finding container 96ad68d03b2bb27af4b8d7309ab2b3262a2abd2f052600483f081f4ca835a6cd: Status 404 returned error can't find the container with id 96ad68d03b2bb27af4b8d7309ab2b3262a2abd2f052600483f081f4ca835a6cd
Sep 05 02:28:15.036531 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:15.036495    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-c26gd" event={"ID":"4caf73f9-59bd-4e2f-ace3-39b25bc5e352","Type":"ContainerStarted","Data":"96ad68d03b2bb27af4b8d7309ab2b3262a2abd2f052600483f081f4ca835a6cd"}
Sep 05 02:28:16.042181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.042140    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-c26gd" event={"ID":"4caf73f9-59bd-4e2f-ace3-39b25bc5e352","Type":"ContainerStarted","Data":"8391557d1554a1c1b85b1a42f4ad24c1d7feb7885c9f544862983594dbfaf7ee"}
Sep 05 02:28:16.060575 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.060525    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-c26gd" podStartSLOduration=1.23798105 podStartE2EDuration="2.060511452s" totalImagesPullingTime="822.530402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:16.058688821 +0000 UTC m=+606.258051814" watchObservedRunningTime="2026-09-05 02:28:16.060511452 +0000 UTC m=+606.259874446"
Sep 05 02:28:16.088886 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.088859    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8t6wq"]
Sep 05 02:28:16.089110 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.089077    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-8t6wq" podUID="49f5861c-3b4a-4690-b935-b5f93ba52177" containerName="authorino" containerID="cri-o://17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309" gracePeriod=30
Sep 05 02:28:16.323383 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.323357    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:28:16.423211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.423176    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5zw8\" (UniqueName: \"kubernetes.io/projected/49f5861c-3b4a-4690-b935-b5f93ba52177-kube-api-access-l5zw8\") pod \"49f5861c-3b4a-4690-b935-b5f93ba52177\" (UID: \"49f5861c-3b4a-4690-b935-b5f93ba52177\") "
Sep 05 02:28:16.425146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.425120    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f5861c-3b4a-4690-b935-b5f93ba52177-kube-api-access-l5zw8" (OuterVolumeSpecName: "kube-api-access-l5zw8") pod "49f5861c-3b4a-4690-b935-b5f93ba52177" (UID: "49f5861c-3b4a-4690-b935-b5f93ba52177"). InnerVolumeSpecName "kube-api-access-l5zw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:28:16.524362 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:16.524330    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5zw8\" (UniqueName: \"kubernetes.io/projected/49f5861c-3b4a-4690-b935-b5f93ba52177-kube-api-access-l5zw8\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:28:17.048249 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.048207    2553 generic.go:356] "Generic (PLEG): container finished" podID="49f5861c-3b4a-4690-b935-b5f93ba52177" containerID="17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309" exitCode=0
Sep 05 02:28:17.048661 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.048267    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8t6wq"
Sep 05 02:28:17.048661 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.048293    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8t6wq" event={"ID":"49f5861c-3b4a-4690-b935-b5f93ba52177","Type":"ContainerDied","Data":"17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309"}
Sep 05 02:28:17.048661 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.048347    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8t6wq" event={"ID":"49f5861c-3b4a-4690-b935-b5f93ba52177","Type":"ContainerDied","Data":"e4a25274b41df95efb881f99b2e5c2abac1429dfc0ab0f4a9367098f53c1533e"}
Sep 05 02:28:17.048661 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.048364    2553 scope.go:117] "RemoveContainer" containerID="17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309"
Sep 05 02:28:17.056458 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.056440    2553 scope.go:117] "RemoveContainer" containerID="17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309"
Sep 05 02:28:17.056755 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:28:17.056736    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309\": container with ID starting with 17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309 not found: ID does not exist" containerID="17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309"
Sep 05 02:28:17.056825 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.056763    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309"} err="failed to get container status \"17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309\": rpc error: code = NotFound desc = could not find container \"17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309\": container with ID starting with 17040b38fd49b3df50ff6741fdd51f7f672a8fe1981e8ffccb614b8055f1f309 not found: ID does not exist"
Sep 05 02:28:17.117274 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.117238    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8t6wq"]
Sep 05 02:28:17.133925 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:17.133898    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8t6wq"]
Sep 05 02:28:18.393332 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:18.393295    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f5861c-3b4a-4690-b935-b5f93ba52177" path="/var/lib/kubelet/pods/49f5861c-3b4a-4690-b935-b5f93ba52177/volumes"
Sep 05 02:28:39.161273 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:39.161236    2553 generic.go:356] "Generic (PLEG): container finished" podID="ce9fdeba-e8a3-4b60-92de-a779b11f760e" containerID="48bee8a1cc16d188e3d8ee05a4ebbb327b46504fa088bdf9e633666fcc1d661b" exitCode=1
Sep 05 02:28:39.161729 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:39.161308    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr" event={"ID":"ce9fdeba-e8a3-4b60-92de-a779b11f760e","Type":"ContainerDied","Data":"48bee8a1cc16d188e3d8ee05a4ebbb327b46504fa088bdf9e633666fcc1d661b"}
Sep 05 02:28:39.161729 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:39.161637    2553 scope.go:117] "RemoveContainer" containerID="48bee8a1cc16d188e3d8ee05a4ebbb327b46504fa088bdf9e633666fcc1d661b"
Sep 05 02:28:40.167532 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:40.167502    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr" event={"ID":"ce9fdeba-e8a3-4b60-92de-a779b11f760e","Type":"ContainerStarted","Data":"62e7dc216d1575d1f0686e71c908c1812fd9b6097e9016c38fbef0d686101395"}
Sep 05 02:28:40.167934 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:40.167734    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:28:51.175121 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:28:51.175091    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-msqjr"
Sep 05 02:32:11.823220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.823180    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"]
Sep 05 02:32:11.823662 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.823552    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f5861c-3b4a-4690-b935-b5f93ba52177" containerName="authorino"
Sep 05 02:32:11.823662 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.823582    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f5861c-3b4a-4690-b935-b5f93ba52177" containerName="authorino"
Sep 05 02:32:11.823662 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.823660    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f5861c-3b4a-4690-b935-b5f93ba52177" containerName="authorino"
Sep 05 02:32:11.826610 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.826594    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.829476 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.829451    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 05 02:32:11.830526 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.830507    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 05 02:32:11.830655 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.830530    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 05 02:32:11.830655 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.830633    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-2prbc\""
Sep 05 02:32:11.836142 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.836115    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"]
Sep 05 02:32:11.883876 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.883853    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-dshm\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.884023 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.883893    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2xr\" (UniqueName: \"kubernetes.io/projected/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-kube-api-access-ns2xr\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.884023 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.883935    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.884023 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.883986    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.884177 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.884032    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-home\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.884177 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.884109    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-dshm\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985106    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2xr\" (UniqueName: \"kubernetes.io/projected/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-kube-api-access-ns2xr\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985132    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985150    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985178    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-home\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985219    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985595    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985687 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985620    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.985687 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.985639    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-home\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.987386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.987362    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-dshm\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.987511 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.987495    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:11.993603 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:11.993581    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2xr\" (UniqueName: \"kubernetes.io/projected/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-kube-api-access-ns2xr\") pod \"fc-smoke-test-kserve-7f4687897-t7mzm\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:12.137686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:12.137613    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:12.409181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:12.409116    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"]
Sep 05 02:32:12.411449 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:32:12.411424    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b8f43b_5cf2_4ac2_ad42_08c1146fcd54.slice/crio-229b0fdd698a5acb40fdf9801dff8a505c2767b2e8a222aa9c536112b4a45dd2 WatchSource:0}: Error finding container 229b0fdd698a5acb40fdf9801dff8a505c2767b2e8a222aa9c536112b4a45dd2: Status 404 returned error can't find the container with id 229b0fdd698a5acb40fdf9801dff8a505c2767b2e8a222aa9c536112b4a45dd2
Sep 05 02:32:12.413699 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:12.413683    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:32:13.171166 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:13.171132    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" event={"ID":"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54","Type":"ContainerStarted","Data":"229b0fdd698a5acb40fdf9801dff8a505c2767b2e8a222aa9c536112b4a45dd2"}
Sep 05 02:32:14.177538 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:14.177498    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" event={"ID":"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54","Type":"ContainerStarted","Data":"be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"}
Sep 05 02:32:14.195502 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:14.195461    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" podStartSLOduration=1.595193527 podStartE2EDuration="3.195445659s" totalImagesPullingTime="1.600252132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:32:14.193752821 +0000 UTC m=+844.393115815" watchObservedRunningTime="2026-09-05 02:32:14.195445659 +0000 UTC m=+844.394808651"
Sep 05 02:32:15.138331 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:15.138292    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:22.138534 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:22.138495    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:22.151094 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:22.151066    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:32:22.223378 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:32:22.223352    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:33:14.937077 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:14.937043    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"]
Sep 05 02:33:14.946845 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:33:14.946810    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:33:14.946845 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:33:14.946845 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:33:14.946845 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60" cmd=["/bin/sleep","15"]
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:14.946868    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" message="" err=<
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:33:14.946883    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerName="main" containerID="cri-o://be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"
Sep 05 02:33:14.947067 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:14.946930    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerName="main" containerID="cri-o://be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60" gracePeriod=60
Sep 05 02:33:15.290340 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.290316    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:33:15.425384 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425359    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tmp-dir\") pod \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") "
Sep 05 02:33:15.425548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425401    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-model-cache\") pod \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") "
Sep 05 02:33:15.425548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425434    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-dshm\") pod \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") "
Sep 05 02:33:15.425548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425460    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tls-certs\") pod \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") "
Sep 05 02:33:15.425548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425484    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-home\") pod \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") "
Sep 05 02:33:15.425548 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425515    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns2xr\" (UniqueName: \"kubernetes.io/projected/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-kube-api-access-ns2xr\") pod \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\" (UID: \"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54\") "
Sep 05 02:33:15.425841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425609    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" (UID: "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:15.425841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425754    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-model-cache" (OuterVolumeSpecName: "model-cache") pod "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" (UID: "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:15.425956 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425845    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-home" (OuterVolumeSpecName: "home") pod "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" (UID: "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:15.425956 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425858    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:33:15.425956 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.425878    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:33:15.427743 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.427721    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" (UID: "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:15.427743 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.427736    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-dshm" (OuterVolumeSpecName: "dshm") pod "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" (UID: "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:15.427884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.427740    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-kube-api-access-ns2xr" (OuterVolumeSpecName: "kube-api-access-ns2xr") pod "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" (UID: "65b8f43b-5cf2-4ac2-ad42-08c1146fcd54"). InnerVolumeSpecName "kube-api-access-ns2xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:15.458614 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.458587    2553 generic.go:356] "Generic (PLEG): container finished" podID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerID="be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60" exitCode=0
Sep 05 02:33:15.458731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.458622    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" event={"ID":"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54","Type":"ContainerDied","Data":"be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"}
Sep 05 02:33:15.458731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.458627    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"
Sep 05 02:33:15.458731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.458660    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" event={"ID":"65b8f43b-5cf2-4ac2-ad42-08c1146fcd54","Type":"ContainerDied","Data":"229b0fdd698a5acb40fdf9801dff8a505c2767b2e8a222aa9c536112b4a45dd2"}
Sep 05 02:33:15.458731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.458682    2553 scope.go:117] "RemoveContainer" containerID="be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"
Sep 05 02:33:15.468103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.468086    2553 scope.go:117] "RemoveContainer" containerID="be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"
Sep 05 02:33:15.468369 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:33:15.468350    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60\": container with ID starting with be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60 not found: ID does not exist" containerID="be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"
Sep 05 02:33:15.468428 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.468375    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60"} err="failed to get container status \"be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60\": rpc error: code = NotFound desc = could not find container \"be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60\": container with ID starting with be635744f65c2db16e1e534c1ef0c72313c1c3b964c7a32164af1e3fffa37f60 not found: ID does not exist"
Sep 05 02:33:15.481039 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.481016    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"]
Sep 05 02:33:15.484304 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.484283    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm"]
Sep 05 02:33:15.527025 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.526998    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:33:15.527025 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.527019    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:33:15.527176 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.527045    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:33:15.527176 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:15.527068    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2xr\" (UniqueName: \"kubernetes.io/projected/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54-kube-api-access-ns2xr\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.213640 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:16.213599    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-t7mzm" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 05 02:33:16.392896 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:33:16.392863    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" path="/var/lib/kubelet/pods/65b8f43b-5cf2-4ac2-ad42-08c1146fcd54/volumes"
Sep 05 02:34:18.011045 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.011008    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"]
Sep 05 02:34:18.011448 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.011402    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerName="main"
Sep 05 02:34:18.011448 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.011414    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerName="main"
Sep 05 02:34:18.011518 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.011482    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b8f43b-5cf2-4ac2-ad42-08c1146fcd54" containerName="main"
Sep 05 02:34:18.015689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.015672    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.019452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.019417    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 05 02:34:18.019452 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.019444    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 05 02:34:18.019690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.019444    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 05 02:34:18.019690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.019426    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-t5vw5\""
Sep 05 02:34:18.024072 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.024049    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"]
Sep 05 02:34:18.134485 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.134442    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-kube-api-access-km4ph\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.134726 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.134504    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-home\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.134726 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.134528    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.134726 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.134652    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.134726 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.134680    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.134726 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.134706    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235140    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235183    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235306    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235426 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235384    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-kube-api-access-km4ph\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235474 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235447    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-home\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235527 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235473    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235663 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235642    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235734 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235711    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.235798 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.235781    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-home\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.237664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.237639    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.237841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.237825    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.244049 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.244029    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-kube-api-access-km4ph\") pod \"fc-concurrency-test-kserve-b5d44467c-79t6q\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.327374 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.327331    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:18.602422 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.602391    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"]
Sep 05 02:34:18.604605 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:34:18.604557    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274daadb_cb86_4c56_bd8a_9ca64bd08ec2.slice/crio-72b88771e744d5be86636b3cf4b42baf0a7557ecfae0489c6e8efd72ca277634 WatchSource:0}: Error finding container 72b88771e744d5be86636b3cf4b42baf0a7557ecfae0489c6e8efd72ca277634: Status 404 returned error can't find the container with id 72b88771e744d5be86636b3cf4b42baf0a7557ecfae0489c6e8efd72ca277634
Sep 05 02:34:18.753515 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.753475    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" event={"ID":"274daadb-cb86-4c56-bd8a-9ca64bd08ec2","Type":"ContainerStarted","Data":"39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"}
Sep 05 02:34:18.753701 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.753538    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" event={"ID":"274daadb-cb86-4c56-bd8a-9ca64bd08ec2","Type":"ContainerStarted","Data":"72b88771e744d5be86636b3cf4b42baf0a7557ecfae0489c6e8efd72ca277634"}
Sep 05 02:34:18.771620 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:18.771526    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" podStartSLOduration=1.771507518 podStartE2EDuration="1.771507518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:34:18.771100872 +0000 UTC m=+968.970463877" watchObservedRunningTime="2026-09-05 02:34:18.771507518 +0000 UTC m=+968.970870512"
Sep 05 02:34:19.328394 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:19.328359    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:28.327626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:28.327587    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:28.339861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:28.339831    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:34:28.812969 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:34:28.812936    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:35:03.540975 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.540932    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"]
Sep 05 02:35:03.544760 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.544731    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.548873 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.548848    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 05 02:35:03.549019 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.548893    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-nckg4\""
Sep 05 02:35:03.549019 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.548908    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 05 02:35:03.549019 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.548937    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-r2g49\""
Sep 05 02:35:03.549173 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.549114    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 05 02:35:03.552384 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.552361    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"]
Sep 05 02:35:03.633088 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.633056    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347e9afd-7d40-4599-bc95-72fa9614163c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.633088 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.633090    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzxj\" (UniqueName: \"kubernetes.io/projected/347e9afd-7d40-4599-bc95-72fa9614163c-kube-api-access-hjzxj\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.734589 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.734524    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347e9afd-7d40-4599-bc95-72fa9614163c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.734758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.734591    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzxj\" (UniqueName: \"kubernetes.io/projected/347e9afd-7d40-4599-bc95-72fa9614163c-kube-api-access-hjzxj\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.737079 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.737054    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347e9afd-7d40-4599-bc95-72fa9614163c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.743754 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.743733    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzxj\" (UniqueName: \"kubernetes.io/projected/347e9afd-7d40-4599-bc95-72fa9614163c-kube-api-access-hjzxj\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:03.856468 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:03.856395    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:04.127179 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:04.127100    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"]
Sep 05 02:35:04.131209 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:35:04.131170    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347e9afd_7d40_4599_bc95_72fa9614163c.slice/crio-2f2d40f737e4aa48f90599df176f8178057c6638b4290e42ff934f558eae8398 WatchSource:0}: Error finding container 2f2d40f737e4aa48f90599df176f8178057c6638b4290e42ff934f558eae8398: Status 404 returned error can't find the container with id 2f2d40f737e4aa48f90599df176f8178057c6638b4290e42ff934f558eae8398
Sep 05 02:35:04.981285 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:04.981247    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" event={"ID":"347e9afd-7d40-4599-bc95-72fa9614163c","Type":"ContainerStarted","Data":"2f2d40f737e4aa48f90599df176f8178057c6638b4290e42ff934f558eae8398"}
Sep 05 02:35:06.512921 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.512883    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"]
Sep 05 02:35:06.522629 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:35:06.522599    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:35:06.522629 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:35:06.522629 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:35:06.522629 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e" cmd=["/bin/sleep","15"]
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.522649    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" message="" err=<
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:35:06.522665    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerName="main" containerID="cri-o://39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"
Sep 05 02:35:06.522810 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.522701    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerName="main" containerID="cri-o://39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e" gracePeriod=60
Sep 05 02:35:06.864455 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.864432    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:35:06.968623 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968594    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-model-cache\") pod \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") "
Sep 05 02:35:06.968783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968661    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tls-certs\") pod \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") "
Sep 05 02:35:06.968783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968700    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tmp-dir\") pod \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") "
Sep 05 02:35:06.968783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968720    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-kube-api-access-km4ph\") pod \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") "
Sep 05 02:35:06.968783 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968770    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-dshm\") pod \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") "
Sep 05 02:35:06.968979 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968801    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-home\") pod \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\" (UID: \"274daadb-cb86-4c56-bd8a-9ca64bd08ec2\") "
Sep 05 02:35:06.968979 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968873    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-model-cache" (OuterVolumeSpecName: "model-cache") pod "274daadb-cb86-4c56-bd8a-9ca64bd08ec2" (UID: "274daadb-cb86-4c56-bd8a-9ca64bd08ec2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:06.968979 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.968945    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "274daadb-cb86-4c56-bd8a-9ca64bd08ec2" (UID: "274daadb-cb86-4c56-bd8a-9ca64bd08ec2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:06.969139 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.969036    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:35:06.969139 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.969071    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:35:06.969139 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.969101    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-home" (OuterVolumeSpecName: "home") pod "274daadb-cb86-4c56-bd8a-9ca64bd08ec2" (UID: "274daadb-cb86-4c56-bd8a-9ca64bd08ec2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:06.970943 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.970917    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-dshm" (OuterVolumeSpecName: "dshm") pod "274daadb-cb86-4c56-bd8a-9ca64bd08ec2" (UID: "274daadb-cb86-4c56-bd8a-9ca64bd08ec2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:06.970943 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.970929    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-kube-api-access-km4ph" (OuterVolumeSpecName: "kube-api-access-km4ph") pod "274daadb-cb86-4c56-bd8a-9ca64bd08ec2" (UID: "274daadb-cb86-4c56-bd8a-9ca64bd08ec2"). InnerVolumeSpecName "kube-api-access-km4ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:06.970943 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.970921    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "274daadb-cb86-4c56-bd8a-9ca64bd08ec2" (UID: "274daadb-cb86-4c56-bd8a-9ca64bd08ec2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:06.993551 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.993518    2553 generic.go:356] "Generic (PLEG): container finished" podID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerID="39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e" exitCode=0
Sep 05 02:35:06.993677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.993581    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"
Sep 05 02:35:06.993677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.993606    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" event={"ID":"274daadb-cb86-4c56-bd8a-9ca64bd08ec2","Type":"ContainerDied","Data":"39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"}
Sep 05 02:35:06.993677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.993662    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" event={"ID":"274daadb-cb86-4c56-bd8a-9ca64bd08ec2","Type":"ContainerDied","Data":"72b88771e744d5be86636b3cf4b42baf0a7557ecfae0489c6e8efd72ca277634"}
Sep 05 02:35:06.993825 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.993693    2553 scope.go:117] "RemoveContainer" containerID="39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"
Sep 05 02:35:06.995325 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.995300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" event={"ID":"347e9afd-7d40-4599-bc95-72fa9614163c","Type":"ContainerStarted","Data":"bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e"}
Sep 05 02:35:06.995464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:06.995350    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:35:07.002536 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.002517    2553 scope.go:117] "RemoveContainer" containerID="39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"
Sep 05 02:35:07.002823 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:35:07.002805    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e\": container with ID starting with 39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e not found: ID does not exist" containerID="39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"
Sep 05 02:35:07.002891 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.002832    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e"} err="failed to get container status \"39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e\": rpc error: code = NotFound desc = could not find container \"39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e\": container with ID starting with 39484dbeb5422a7ab858942386d1121e8d8c2c007027c7c77ad7a9026a45932e not found: ID does not exist"
Sep 05 02:35:07.014791 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.014748    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podStartSLOduration=1.626947436 podStartE2EDuration="4.014734886s" totalImagesPullingTime="2.38778745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:07.013790306 +0000 UTC m=+1017.213153301" watchObservedRunningTime="2026-09-05 02:35:07.014734886 +0000 UTC m=+1017.214097878"
Sep 05 02:35:07.027550 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.027525    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"]
Sep 05 02:35:07.031178 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.031158    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q"]
Sep 05 02:35:07.070251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.070178    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:35:07.070251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.070202    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:35:07.070251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.070213    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-kube-api-access-km4ph\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:35:07.070251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.070226    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274daadb-cb86-4c56-bd8a-9ca64bd08ec2-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:35:07.802670 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:07.802629    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-79t6q" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": context deadline exceeded"
Sep 05 02:35:08.392999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:08.392964    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" path="/var/lib/kubelet/pods/274daadb-cb86-4c56-bd8a-9ca64bd08ec2/volumes"
Sep 05 02:35:38.003235 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:35:38.003201    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:36:47.327431 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.327395    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"]
Sep 05 02:36:47.327866 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.327801    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerName="main"
Sep 05 02:36:47.327866 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.327814    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerName="main"
Sep 05 02:36:47.327939 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.327887    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="274daadb-cb86-4c56-bd8a-9ca64bd08ec2" containerName="main"
Sep 05 02:36:47.331163 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.331143    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.334936 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.334912    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-xt5gh\""
Sep 05 02:36:47.334936 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.334929    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 05 02:36:47.335168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.334987    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-26dwd\""
Sep 05 02:36:47.335168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.334936    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 05 02:36:47.335168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.335117    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 05 02:36:47.347249 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.347226    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"]
Sep 05 02:36:47.405728 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.405698    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8f9\" (UniqueName: \"kubernetes.io/projected/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-kube-api-access-vp8f9\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.405889 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.405759    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.506313 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.506279    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.506512 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.506401    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8f9\" (UniqueName: \"kubernetes.io/projected/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-kube-api-access-vp8f9\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.508887 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.508867    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.514924 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.514897    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8f9\" (UniqueName: \"kubernetes.io/projected/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-kube-api-access-vp8f9\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.641541 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.641463    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:47.915246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:47.915168    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"]
Sep 05 02:36:47.918231 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:36:47.918207    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bb00cb_6cc1_4f78_a2df_ff0ce66f17d7.slice/crio-3d31e77383f5a90e8373ada4e649d38804b37cbea282ae6670dfc7a2846fdc59 WatchSource:0}: Error finding container 3d31e77383f5a90e8373ada4e649d38804b37cbea282ae6670dfc7a2846fdc59: Status 404 returned error can't find the container with id 3d31e77383f5a90e8373ada4e649d38804b37cbea282ae6670dfc7a2846fdc59
Sep 05 02:36:48.488787 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:48.488740    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" event={"ID":"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7","Type":"ContainerStarted","Data":"d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45"}
Sep 05 02:36:48.489241 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:48.488806    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" event={"ID":"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7","Type":"ContainerStarted","Data":"3d31e77383f5a90e8373ada4e649d38804b37cbea282ae6670dfc7a2846fdc59"}
Sep 05 02:36:48.489241 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:48.488888    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:36:48.507666 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:36:48.507613    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" podStartSLOduration=1.5075945750000002 podStartE2EDuration="1.507594575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:36:48.507163163 +0000 UTC m=+1118.706526155" watchObservedRunningTime="2026-09-05 02:36:48.507594575 +0000 UTC m=+1118.706957565"
Sep 05 02:37:17.433214 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:17.433171    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"]
Sep 05 02:37:18.003640 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:18.003601    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:37:19.497341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:19.497308    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:37:28.003430 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:28.003390    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:37:32.454672 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:32.454625    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main" containerID="cri-o://bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e" gracePeriod=45
Sep 05 02:37:38.003041 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:38.002939    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:37:38.003458 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:38.003068    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:37:48.002907 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:48.002870    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:37:58.002716 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:37:58.002678    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:38:02.705388 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.705363    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:38:02.736034 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.735966    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347e9afd-7d40-4599-bc95-72fa9614163c-tls-certs\") pod \"347e9afd-7d40-4599-bc95-72fa9614163c\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") "
Sep 05 02:38:02.736164 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.736080    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjzxj\" (UniqueName: \"kubernetes.io/projected/347e9afd-7d40-4599-bc95-72fa9614163c-kube-api-access-hjzxj\") pod \"347e9afd-7d40-4599-bc95-72fa9614163c\" (UID: \"347e9afd-7d40-4599-bc95-72fa9614163c\") "
Sep 05 02:38:02.738271 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.738237    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347e9afd-7d40-4599-bc95-72fa9614163c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "347e9afd-7d40-4599-bc95-72fa9614163c" (UID: "347e9afd-7d40-4599-bc95-72fa9614163c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:38:02.738381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.738298    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347e9afd-7d40-4599-bc95-72fa9614163c-kube-api-access-hjzxj" (OuterVolumeSpecName: "kube-api-access-hjzxj") pod "347e9afd-7d40-4599-bc95-72fa9614163c" (UID: "347e9afd-7d40-4599-bc95-72fa9614163c"). InnerVolumeSpecName "kube-api-access-hjzxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:38:02.837717 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.837691    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjzxj\" (UniqueName: \"kubernetes.io/projected/347e9afd-7d40-4599-bc95-72fa9614163c-kube-api-access-hjzxj\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:38:02.837717 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.837716    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347e9afd-7d40-4599-bc95-72fa9614163c-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:38:02.843742 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.843715    2553 generic.go:356] "Generic (PLEG): container finished" podID="347e9afd-7d40-4599-bc95-72fa9614163c" containerID="bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e" exitCode=0
Sep 05 02:38:02.843852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.843763    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"
Sep 05 02:38:02.843852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.843765    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" event={"ID":"347e9afd-7d40-4599-bc95-72fa9614163c","Type":"ContainerDied","Data":"bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e"}
Sep 05 02:38:02.843957 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.843886    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq" event={"ID":"347e9afd-7d40-4599-bc95-72fa9614163c","Type":"ContainerDied","Data":"2f2d40f737e4aa48f90599df176f8178057c6638b4290e42ff934f558eae8398"}
Sep 05 02:38:02.843957 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.843915    2553 scope.go:117] "RemoveContainer" containerID="bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e"
Sep 05 02:38:02.852593 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.852552    2553 scope.go:117] "RemoveContainer" containerID="bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e"
Sep 05 02:38:02.852829 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:38:02.852812    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e\": container with ID starting with bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e not found: ID does not exist" containerID="bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e"
Sep 05 02:38:02.852888 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.852836    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e"} err="failed to get container status \"bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e\": rpc error: code = NotFound desc = could not find container \"bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e\": container with ID starting with bd035eaeb354bece7663bb3f9c869274171581bedb36499da3435411f371c62e not found: ID does not exist"
Sep 05 02:38:02.870747 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.870724    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"]
Sep 05 02:38:02.876581 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:02.872783    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexk2mq"]
Sep 05 02:38:04.393074 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:04.393042    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" path="/var/lib/kubelet/pods/347e9afd-7d40-4599-bc95-72fa9614163c/volumes"
Sep 05 02:38:29.507975 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.507944    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"]
Sep 05 02:38:29.508586 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.508510    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main"
Sep 05 02:38:29.508586 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.508530    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main"
Sep 05 02:38:29.508726 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.508641    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="347e9afd-7d40-4599-bc95-72fa9614163c" containerName="main"
Sep 05 02:38:29.519628 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.519601    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.522073 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.522049    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"]
Sep 05 02:38:29.523462 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.523440    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 05 02:38:29.523595 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.523493    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 05 02:38:29.523595 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.523550    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 05 02:38:29.523717 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.523655    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-dq9g7\""
Sep 05 02:38:29.659394 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.659366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.659549 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.659432    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.659549 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.659510    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.659690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.659551    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ee813e-ff5e-4058-8a44-38985188db4c-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.659690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.659625    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.659690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.659644    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjnm\" (UniqueName: \"kubernetes.io/projected/b1ee813e-ff5e-4058-8a44-38985188db4c-kube-api-access-jrjnm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.760454 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760388    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.760454 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760451    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.760676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760485    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.760676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ee813e-ff5e-4058-8a44-38985188db4c-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.760676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760538    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.761000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760870    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.761000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760887    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.761000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760927    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjnm\" (UniqueName: \"kubernetes.io/projected/b1ee813e-ff5e-4058-8a44-38985188db4c-kube-api-access-jrjnm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.761000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.760956    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.762679 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.762657    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.763122 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.763102    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ee813e-ff5e-4058-8a44-38985188db4c-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.772296 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.772271    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjnm\" (UniqueName: \"kubernetes.io/projected/b1ee813e-ff5e-4058-8a44-38985188db4c-kube-api-access-jrjnm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d99kh\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:29.831388 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:29.831363    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:30.107583 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:30.107547    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"]
Sep 05 02:38:30.109741 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:38:30.109711    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ee813e_ff5e_4058_8a44_38985188db4c.slice/crio-4c576c3a77a6cf137a279d7029e9927d3dd401c16f040c9657801ee784f21aba WatchSource:0}: Error finding container 4c576c3a77a6cf137a279d7029e9927d3dd401c16f040c9657801ee784f21aba: Status 404 returned error can't find the container with id 4c576c3a77a6cf137a279d7029e9927d3dd401c16f040c9657801ee784f21aba
Sep 05 02:38:30.982151 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:30.982111    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" event={"ID":"b1ee813e-ff5e-4058-8a44-38985188db4c","Type":"ContainerStarted","Data":"30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"}
Sep 05 02:38:30.982501 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:30.982170    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" event={"ID":"b1ee813e-ff5e-4058-8a44-38985188db4c","Type":"ContainerStarted","Data":"4c576c3a77a6cf137a279d7029e9927d3dd401c16f040c9657801ee784f21aba"}
Sep 05 02:38:31.008219 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:31.008158    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" podStartSLOduration=2.008137647 podStartE2EDuration="2.008137647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:31.005529899 +0000 UTC m=+1221.204892889" watchObservedRunningTime="2026-09-05 02:38:31.008137647 +0000 UTC m=+1221.207500641"
Sep 05 02:38:31.832149 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:31.832108    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:39.832468 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:39.832438    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:39.845793 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:39.845763    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:40.036282 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:40.036251    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:38:52.176598 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:52.176549    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"]
Sep 05 02:38:59.497025 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:38:59.496985    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:03.737292 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:03.737214    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"]
Sep 05 02:39:03.748113 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:39:03.748075    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:39:03.748113 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:39:03.748113 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:39:03.748113 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a" cmd=["/bin/sleep","15"]
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:03.748130    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" message="" err=<
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:39:03.748142    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerName="main" containerID="cri-o://30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"
Sep 05 02:39:03.748413 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:03.748190    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerName="main" containerID="cri-o://30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a" gracePeriod=60
Sep 05 02:39:04.095593 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.095550    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:39:04.139884 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.139858    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-model-cache\") pod \"b1ee813e-ff5e-4058-8a44-38985188db4c\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") "
Sep 05 02:39:04.140048 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.139911    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrjnm\" (UniqueName: \"kubernetes.io/projected/b1ee813e-ff5e-4058-8a44-38985188db4c-kube-api-access-jrjnm\") pod \"b1ee813e-ff5e-4058-8a44-38985188db4c\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") "
Sep 05 02:39:04.140048 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.139944    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-dshm\") pod \"b1ee813e-ff5e-4058-8a44-38985188db4c\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") "
Sep 05 02:39:04.140048 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.139980    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-home\") pod \"b1ee813e-ff5e-4058-8a44-38985188db4c\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") "
Sep 05 02:39:04.140229 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140135    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1ee813e-ff5e-4058-8a44-38985188db4c" (UID: "b1ee813e-ff5e-4058-8a44-38985188db4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:04.140229 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140186    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-home" (OuterVolumeSpecName: "home") pod "b1ee813e-ff5e-4058-8a44-38985188db4c" (UID: "b1ee813e-ff5e-4058-8a44-38985188db4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:04.140229 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140204    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-tmp-dir\") pod \"b1ee813e-ff5e-4058-8a44-38985188db4c\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") "
Sep 05 02:39:04.140379 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140342    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ee813e-ff5e-4058-8a44-38985188db4c-tls-certs\") pod \"b1ee813e-ff5e-4058-8a44-38985188db4c\" (UID: \"b1ee813e-ff5e-4058-8a44-38985188db4c\") "
Sep 05 02:39:04.140492 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140460    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1ee813e-ff5e-4058-8a44-38985188db4c" (UID: "b1ee813e-ff5e-4058-8a44-38985188db4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:04.140678 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140659    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:04.140781 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140681    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:04.140781 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.140693    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:04.141205 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.141182    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerID="30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a" exitCode=0
Sep 05 02:39:04.141303 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.141245    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" event={"ID":"b1ee813e-ff5e-4058-8a44-38985188db4c","Type":"ContainerDied","Data":"30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"}
Sep 05 02:39:04.141303 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.141252    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"
Sep 05 02:39:04.141303 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.141294    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" event={"ID":"b1ee813e-ff5e-4058-8a44-38985188db4c","Type":"ContainerDied","Data":"4c576c3a77a6cf137a279d7029e9927d3dd401c16f040c9657801ee784f21aba"}
Sep 05 02:39:04.141462 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.141323    2553 scope.go:117] "RemoveContainer" containerID="30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"
Sep 05 02:39:04.142672 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.142646    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ee813e-ff5e-4058-8a44-38985188db4c-kube-api-access-jrjnm" (OuterVolumeSpecName: "kube-api-access-jrjnm") pod "b1ee813e-ff5e-4058-8a44-38985188db4c" (UID: "b1ee813e-ff5e-4058-8a44-38985188db4c"). InnerVolumeSpecName "kube-api-access-jrjnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:39:04.142797 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.142774    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ee813e-ff5e-4058-8a44-38985188db4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1ee813e-ff5e-4058-8a44-38985188db4c" (UID: "b1ee813e-ff5e-4058-8a44-38985188db4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:39:04.142955 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.142933    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-dshm" (OuterVolumeSpecName: "dshm") pod "b1ee813e-ff5e-4058-8a44-38985188db4c" (UID: "b1ee813e-ff5e-4058-8a44-38985188db4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:04.165357 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.165339    2553 scope.go:117] "RemoveContainer" containerID="30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"
Sep 05 02:39:04.165628 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:39:04.165608    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a\": container with ID starting with 30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a not found: ID does not exist" containerID="30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"
Sep 05 02:39:04.165690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.165636    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a"} err="failed to get container status \"30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a\": rpc error: code = NotFound desc = could not find container \"30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a\": container with ID starting with 30703f8fc73e208e0ad48f1da8508d9753999ae2e2845e283dc0db438e4c201a not found: ID does not exist"
Sep 05 02:39:04.241185 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.241162    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ee813e-ff5e-4058-8a44-38985188db4c-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:04.241185 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.241185    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjnm\" (UniqueName: \"kubernetes.io/projected/b1ee813e-ff5e-4058-8a44-38985188db4c-kube-api-access-jrjnm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:04.241304 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.241195    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ee813e-ff5e-4058-8a44-38985188db4c-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:04.457141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.457112    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"]
Sep 05 02:39:04.459816 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:04.459794    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh"]
Sep 05 02:39:05.025731 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:05.025694    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d99kh" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded"
Sep 05 02:39:06.393717 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:06.393683    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" path="/var/lib/kubelet/pods/b1ee813e-ff5e-4058-8a44-38985188db4c/volumes"
Sep 05 02:39:07.197378 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:07.197340    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main" containerID="cri-o://d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45" gracePeriod=45
Sep 05 02:39:09.497817 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:09.497781    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:19.496976 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:19.496936    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:19.497343 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:19.497025    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:39:29.497424 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:29.497388    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:37.448999 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.448976    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:39:37.614626 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.614599    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp8f9\" (UniqueName: \"kubernetes.io/projected/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-kube-api-access-vp8f9\") pod \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") "
Sep 05 02:39:37.614784 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.614679    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-tls-certs\") pod \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\" (UID: \"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7\") "
Sep 05 02:39:37.616764 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.616737    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-kube-api-access-vp8f9" (OuterVolumeSpecName: "kube-api-access-vp8f9") pod "13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" (UID: "13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7"). InnerVolumeSpecName "kube-api-access-vp8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:39:37.616889 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.616775    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" (UID: "13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:39:37.715715 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.715682    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8f9\" (UniqueName: \"kubernetes.io/projected/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-kube-api-access-vp8f9\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:37.715715 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:37.715711    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:39:38.303465 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.303429    2553 generic.go:356] "Generic (PLEG): container finished" podID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerID="d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45" exitCode=0
Sep 05 02:39:38.303664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.303473    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"
Sep 05 02:39:38.303664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.303497    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" event={"ID":"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7","Type":"ContainerDied","Data":"d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45"}
Sep 05 02:39:38.303664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.303540    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv" event={"ID":"13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7","Type":"ContainerDied","Data":"3d31e77383f5a90e8373ada4e649d38804b37cbea282ae6670dfc7a2846fdc59"}
Sep 05 02:39:38.303664 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.303581    2553 scope.go:117] "RemoveContainer" containerID="d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45"
Sep 05 02:39:38.312412 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.312391    2553 scope.go:117] "RemoveContainer" containerID="d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45"
Sep 05 02:39:38.312683 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:39:38.312663    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45\": container with ID starting with d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45 not found: ID does not exist" containerID="d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45"
Sep 05 02:39:38.312738 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.312689    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45"} err="failed to get container status \"d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45\": rpc error: code = NotFound desc = could not find container \"d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45\": container with ID starting with d8e0c4c57a2971f2bda92ec914e0232f70462d83c63ee647b8e158fbd9207d45 not found: ID does not exist"
Sep 05 02:39:38.324876 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.324829    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"]
Sep 05 02:39:38.326718 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.326698    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4drfv"]
Sep 05 02:39:38.393469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:38.393436    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" path="/var/lib/kubelet/pods/13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7/volumes"
Sep 05 02:39:55.280774 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.280740    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"]
Sep 05 02:39:55.281137 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.281115    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerName="main"
Sep 05 02:39:55.281137 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.281128    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerName="main"
Sep 05 02:39:55.281211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.281157    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main"
Sep 05 02:39:55.281211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.281164    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main"
Sep 05 02:39:55.281283 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.281244    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="13bb00cb-6cc1-4f78-a2df-ff0ce66f17d7" containerName="main"
Sep 05 02:39:55.281283 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.281253    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ee813e-ff5e-4058-8a44-38985188db4c" containerName="main"
Sep 05 02:39:55.284234 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.284218    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.287863 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.287837    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 05 02:39:55.288009 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.287900    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 05 02:39:55.288009 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.287910    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-rmq52\""
Sep 05 02:39:55.288193 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.288176    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-mxmcd\""
Sep 05 02:39:55.288266 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.288211    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 05 02:39:55.291857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.291835    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"]
Sep 05 02:39:55.346327 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.346305    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqn9\" (UniqueName: \"kubernetes.io/projected/4120da62-15ba-422a-9ae3-e4dc5831f03b-kube-api-access-tzqn9\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.346430 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.346363    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120da62-15ba-422a-9ae3-e4dc5831f03b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.447003 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.446974    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120da62-15ba-422a-9ae3-e4dc5831f03b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.447144 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.447052    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqn9\" (UniqueName: \"kubernetes.io/projected/4120da62-15ba-422a-9ae3-e4dc5831f03b-kube-api-access-tzqn9\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.449596 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.449552    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120da62-15ba-422a-9ae3-e4dc5831f03b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.462073 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.462047    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqn9\" (UniqueName: \"kubernetes.io/projected/4120da62-15ba-422a-9ae3-e4dc5831f03b-kube-api-access-tzqn9\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.594857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.594793    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:55.869676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:55.869583    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"]
Sep 05 02:39:55.872778 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:39:55.872747    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4120da62_15ba_422a_9ae3_e4dc5831f03b.slice/crio-d9d6bfe6e438b0a25f9e1ccf44b0dec7d10f59b7df7d23ab281fbac6440f0c70 WatchSource:0}: Error finding container d9d6bfe6e438b0a25f9e1ccf44b0dec7d10f59b7df7d23ab281fbac6440f0c70: Status 404 returned error can't find the container with id d9d6bfe6e438b0a25f9e1ccf44b0dec7d10f59b7df7d23ab281fbac6440f0c70
Sep 05 02:39:56.395870 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:56.395837    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" event={"ID":"4120da62-15ba-422a-9ae3-e4dc5831f03b","Type":"ContainerStarted","Data":"ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340"}
Sep 05 02:39:56.396242 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:56.395883    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" event={"ID":"4120da62-15ba-422a-9ae3-e4dc5831f03b","Type":"ContainerStarted","Data":"d9d6bfe6e438b0a25f9e1ccf44b0dec7d10f59b7df7d23ab281fbac6440f0c70"}
Sep 05 02:39:56.396242 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:56.395918    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:39:56.414470 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:39:56.414421    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" podStartSLOduration=1.41440381 podStartE2EDuration="1.41440381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:39:56.412369705 +0000 UTC m=+1306.611732723" watchObservedRunningTime="2026-09-05 02:39:56.41440381 +0000 UTC m=+1306.613766803"
Sep 05 02:40:05.798530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.798494    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"]
Sep 05 02:40:05.802232 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.802208    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:05.806291 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.806265    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-x27zz\""
Sep 05 02:40:05.806424 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.806265    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 05 02:40:05.806424 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.806275    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 05 02:40:05.806545 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.806348    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 05 02:40:05.814190 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.814166    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"]
Sep 05 02:40:05.936727 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.936692    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e564b615-aa0a-48d2-9acf-9b57c920f816-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:05.936727 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.936731    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:05.936898 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.936758    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:05.936898 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.936829    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:05.936898 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.936858    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdr6\" (UniqueName: \"kubernetes.io/projected/e564b615-aa0a-48d2-9acf-9b57c920f816-kube-api-access-msdr6\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:05.936898 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:05.936882    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.037696 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.037667    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.037834 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.037712    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.037918 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.037839    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.037918 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.037887    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdr6\" (UniqueName: \"kubernetes.io/projected/e564b615-aa0a-48d2-9acf-9b57c920f816-kube-api-access-msdr6\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.038030 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.037925    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.038030 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.038025    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e564b615-aa0a-48d2-9acf-9b57c920f816-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.038133 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.038110    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.038199 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.038165    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.038313 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.038293    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.040239 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.040214    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.040434 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.040416    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e564b615-aa0a-48d2-9acf-9b57c920f816-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.049583 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.049501    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdr6\" (UniqueName: \"kubernetes.io/projected/e564b615-aa0a-48d2-9acf-9b57c920f816-kube-api-access-msdr6\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.114495 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.114469    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:06.394666 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:40:06.394635    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode564b615_aa0a_48d2_9acf_9b57c920f816.slice/crio-f27ccd7b0aa969179145f4738267a0435bdde413fa1795e543cd176d6e42f096 WatchSource:0}: Error finding container f27ccd7b0aa969179145f4738267a0435bdde413fa1795e543cd176d6e42f096: Status 404 returned error can't find the container with id f27ccd7b0aa969179145f4738267a0435bdde413fa1795e543cd176d6e42f096
Sep 05 02:40:06.396693 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.396662    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"]
Sep 05 02:40:06.445312 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:06.445286    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" event={"ID":"e564b615-aa0a-48d2-9acf-9b57c920f816","Type":"ContainerStarted","Data":"f27ccd7b0aa969179145f4738267a0435bdde413fa1795e543cd176d6e42f096"}
Sep 05 02:40:07.451646 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:07.451607    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" event={"ID":"e564b615-aa0a-48d2-9acf-9b57c920f816","Type":"ContainerStarted","Data":"bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5"}
Sep 05 02:40:07.470198 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:07.470025    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" podStartSLOduration=2.470011391 podStartE2EDuration="2.470011391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:07.468781832 +0000 UTC m=+1317.668144826" watchObservedRunningTime="2026-09-05 02:40:07.470011391 +0000 UTC m=+1317.669374384"
Sep 05 02:40:08.115037 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:08.115001    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:16.115293 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:16.115259    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:16.127520 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:16.127493    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:16.505378 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:16.505295    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:27.403216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:27.403188    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:40:39.345105 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.344903    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"]
Sep 05 02:40:39.355493 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:40:39.355450    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:40:39.355493 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:40:39.355493 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:40:39.355493 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5" cmd=["/bin/sleep","15"]
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.355514    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" message="" err=<
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:40:39.355531    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerName="main" containerID="cri-o://bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5"
Sep 05 02:40:39.355848 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.355607    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerName="main" containerID="cri-o://bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5" gracePeriod=60
Sep 05 02:40:39.495087 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.495054    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 05 02:40:39.605186 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.605105    2553 generic.go:356] "Generic (PLEG): container finished" podID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerID="bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5" exitCode=0
Sep 05 02:40:39.605186 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.605175    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" event={"ID":"e564b615-aa0a-48d2-9acf-9b57c920f816","Type":"ContainerDied","Data":"bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5"}
Sep 05 02:40:39.708160 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.708138    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:39.719507 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719481    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e564b615-aa0a-48d2-9acf-9b57c920f816-tls-certs\") pod \"e564b615-aa0a-48d2-9acf-9b57c920f816\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") "
Sep 05 02:40:39.719658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719517    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-tmp-dir\") pod \"e564b615-aa0a-48d2-9acf-9b57c920f816\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") "
Sep 05 02:40:39.719658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719574    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-model-cache\") pod \"e564b615-aa0a-48d2-9acf-9b57c920f816\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") "
Sep 05 02:40:39.719658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719602    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-home\") pod \"e564b615-aa0a-48d2-9acf-9b57c920f816\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") "
Sep 05 02:40:39.719658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719615    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-dshm\") pod \"e564b615-aa0a-48d2-9acf-9b57c920f816\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") "
Sep 05 02:40:39.719658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719649    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msdr6\" (UniqueName: \"kubernetes.io/projected/e564b615-aa0a-48d2-9acf-9b57c920f816-kube-api-access-msdr6\") pod \"e564b615-aa0a-48d2-9acf-9b57c920f816\" (UID: \"e564b615-aa0a-48d2-9acf-9b57c920f816\") "
Sep 05 02:40:39.719955 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719789    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e564b615-aa0a-48d2-9acf-9b57c920f816" (UID: "e564b615-aa0a-48d2-9acf-9b57c920f816"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:39.719955 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719870    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-home" (OuterVolumeSpecName: "home") pod "e564b615-aa0a-48d2-9acf-9b57c920f816" (UID: "e564b615-aa0a-48d2-9acf-9b57c920f816"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:39.719955 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719878    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-model-cache" (OuterVolumeSpecName: "model-cache") pod "e564b615-aa0a-48d2-9acf-9b57c920f816" (UID: "e564b615-aa0a-48d2-9acf-9b57c920f816"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:39.719955 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.719918    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:40:39.721636 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.721607    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e564b615-aa0a-48d2-9acf-9b57c920f816-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e564b615-aa0a-48d2-9acf-9b57c920f816" (UID: "e564b615-aa0a-48d2-9acf-9b57c920f816"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:39.721765 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.721733    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e564b615-aa0a-48d2-9acf-9b57c920f816-kube-api-access-msdr6" (OuterVolumeSpecName: "kube-api-access-msdr6") pod "e564b615-aa0a-48d2-9acf-9b57c920f816" (UID: "e564b615-aa0a-48d2-9acf-9b57c920f816"). InnerVolumeSpecName "kube-api-access-msdr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:39.721765 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.721734    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-dshm" (OuterVolumeSpecName: "dshm") pod "e564b615-aa0a-48d2-9acf-9b57c920f816" (UID: "e564b615-aa0a-48d2-9acf-9b57c920f816"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:39.821022 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.820997    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msdr6\" (UniqueName: \"kubernetes.io/projected/e564b615-aa0a-48d2-9acf-9b57c920f816-kube-api-access-msdr6\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:40:39.821022 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.821019    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e564b615-aa0a-48d2-9acf-9b57c920f816-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:40:39.821167 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.821029    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:40:39.821167 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.821038    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:40:39.821167 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:39.821045    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e564b615-aa0a-48d2-9acf-9b57c920f816-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:40:40.610496 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:40.610466    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"
Sep 05 02:40:40.610947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:40.610442    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl" event={"ID":"e564b615-aa0a-48d2-9acf-9b57c920f816","Type":"ContainerDied","Data":"f27ccd7b0aa969179145f4738267a0435bdde413fa1795e543cd176d6e42f096"}
Sep 05 02:40:40.610947 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:40.610623    2553 scope.go:117] "RemoveContainer" containerID="bca479a9aa2c4dc21fae75cf13cef16b8a582f989c7c46b198dfbfeb8a7968d5"
Sep 05 02:40:40.626927 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:40.626900    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"]
Sep 05 02:40:40.632730 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:40.632704    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fx6trl"]
Sep 05 02:40:42.392648 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:40:42.392607    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" path="/var/lib/kubelet/pods/e564b615-aa0a-48d2-9acf-9b57c920f816/volumes"
Sep 05 02:41:40.226215 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.226179    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"]
Sep 05 02:41:40.227027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.226571    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerName="main"
Sep 05 02:41:40.227027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.226585    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerName="main"
Sep 05 02:41:40.227027 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.226652    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564b615-aa0a-48d2-9acf-9b57c920f816" containerName="main"
Sep 05 02:41:40.231124 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.231073    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.234870 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.234846    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-hxlrf\""
Sep 05 02:41:40.234870 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.234871    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 05 02:41:40.235076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.234928    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 05 02:41:40.235076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.234976    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-bzj8h\""
Sep 05 02:41:40.235220 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.235196    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 05 02:41:40.237985 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.237962    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"]
Sep 05 02:41:40.345435 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.345404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wd9\" (UniqueName: \"kubernetes.io/projected/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-kube-api-access-98wd9\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.345675 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.345451    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.446774 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.446744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.446954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.446839    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wd9\" (UniqueName: \"kubernetes.io/projected/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-kube-api-access-98wd9\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.449442 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.449421    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.455210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.455191    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wd9\" (UniqueName: \"kubernetes.io/projected/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-kube-api-access-98wd9\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.543692 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.543670    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.819707 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.819679    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"]
Sep 05 02:41:40.821608 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:41:40.821581    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f5d2ab_9509_4cfa_a0bd_3abeccf29ff4.slice/crio-a3162c1bdb634c8227eefd6fac0c71e84f1340adb65e266009a586c950974c58 WatchSource:0}: Error finding container a3162c1bdb634c8227eefd6fac0c71e84f1340adb65e266009a586c950974c58: Status 404 returned error can't find the container with id a3162c1bdb634c8227eefd6fac0c71e84f1340adb65e266009a586c950974c58
Sep 05 02:41:40.914079 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.914044    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" event={"ID":"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4","Type":"ContainerStarted","Data":"655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba"}
Sep 05 02:41:40.914258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.914101    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" event={"ID":"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4","Type":"ContainerStarted","Data":"a3162c1bdb634c8227eefd6fac0c71e84f1340adb65e266009a586c950974c58"}
Sep 05 02:41:40.914258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.914123    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:41:40.933067 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:40.933012    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" podStartSLOduration=0.932997645 podStartE2EDuration="932.997645ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:40.931049173 +0000 UTC m=+1411.130412163" watchObservedRunningTime="2026-09-05 02:41:40.932997645 +0000 UTC m=+1411.132360637"
Sep 05 02:41:57.482538 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:41:57.482482    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"]
Sep 05 02:42:07.403666 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:07.403556    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:11.921072 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:11.921042    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:42:12.502112 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:12.502076    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main" containerID="cri-o://ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340" gracePeriod=45
Sep 05 02:42:13.549062 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:13.549026    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"]
Sep 05 02:42:17.402968 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:17.402926    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:21.921708 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:21.921661    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:27.403107 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:27.403063    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:27.403482 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:27.403187    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:42:28.569085 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:28.569040    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main" containerID="cri-o://655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba" gracePeriod=45
Sep 05 02:42:31.921386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:31.921348    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:37.403571 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:37.403530    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:41.921007 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:41.920963    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:41.921377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:41.921079    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:42:42.737753 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.737732    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:42:42.770175 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.770117    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzqn9\" (UniqueName: \"kubernetes.io/projected/4120da62-15ba-422a-9ae3-e4dc5831f03b-kube-api-access-tzqn9\") pod \"4120da62-15ba-422a-9ae3-e4dc5831f03b\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") "
Sep 05 02:42:42.770175 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.770168    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120da62-15ba-422a-9ae3-e4dc5831f03b-tls-certs\") pod \"4120da62-15ba-422a-9ae3-e4dc5831f03b\" (UID: \"4120da62-15ba-422a-9ae3-e4dc5831f03b\") "
Sep 05 02:42:42.772300 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.772277    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4120da62-15ba-422a-9ae3-e4dc5831f03b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4120da62-15ba-422a-9ae3-e4dc5831f03b" (UID: "4120da62-15ba-422a-9ae3-e4dc5831f03b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:42.772399 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.772292    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4120da62-15ba-422a-9ae3-e4dc5831f03b-kube-api-access-tzqn9" (OuterVolumeSpecName: "kube-api-access-tzqn9") pod "4120da62-15ba-422a-9ae3-e4dc5831f03b" (UID: "4120da62-15ba-422a-9ae3-e4dc5831f03b"). InnerVolumeSpecName "kube-api-access-tzqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:42.871246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.871214    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzqn9\" (UniqueName: \"kubernetes.io/projected/4120da62-15ba-422a-9ae3-e4dc5831f03b-kube-api-access-tzqn9\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:42:42.871246 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:42.871241    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4120da62-15ba-422a-9ae3-e4dc5831f03b-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:42:43.210836 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.210798    2553 generic.go:356] "Generic (PLEG): container finished" podID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerID="ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340" exitCode=0
Sep 05 02:42:43.211216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.210842    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"
Sep 05 02:42:43.211216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.210890    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" event={"ID":"4120da62-15ba-422a-9ae3-e4dc5831f03b","Type":"ContainerDied","Data":"ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340"}
Sep 05 02:42:43.211216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.210950    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr" event={"ID":"4120da62-15ba-422a-9ae3-e4dc5831f03b","Type":"ContainerDied","Data":"d9d6bfe6e438b0a25f9e1ccf44b0dec7d10f59b7df7d23ab281fbac6440f0c70"}
Sep 05 02:42:43.211216 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.210977    2553 scope.go:117] "RemoveContainer" containerID="ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340"
Sep 05 02:42:43.219620 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.219604    2553 scope.go:117] "RemoveContainer" containerID="ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340"
Sep 05 02:42:43.219875 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:42:43.219857    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340\": container with ID starting with ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340 not found: ID does not exist" containerID="ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340"
Sep 05 02:42:43.219945 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.219889    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340"} err="failed to get container status \"ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340\": rpc error: code = NotFound desc = could not find container \"ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340\": container with ID starting with ae73f3154aefc6d26ef57b285963f7bc1acc293b0e481f45a7fe87048b54f340 not found: ID does not exist"
Sep 05 02:42:43.232299 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.232278    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"]
Sep 05 02:42:43.239025 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:43.239004    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cpx7nr"]
Sep 05 02:42:44.395593 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:44.395545    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" path="/var/lib/kubelet/pods/4120da62-15ba-422a-9ae3-e4dc5831f03b/volumes"
Sep 05 02:42:51.921088 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:51.921043    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:42:58.826924 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:58.826854    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:42:58.905738 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:58.905712    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98wd9\" (UniqueName: \"kubernetes.io/projected/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-kube-api-access-98wd9\") pod \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") "
Sep 05 02:42:58.905876 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:58.905831    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-tls-certs\") pod \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\" (UID: \"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4\") "
Sep 05 02:42:58.907787 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:58.907762    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-kube-api-access-98wd9" (OuterVolumeSpecName: "kube-api-access-98wd9") pod "e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" (UID: "e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4"). InnerVolumeSpecName "kube-api-access-98wd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:58.907879 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:58.907818    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" (UID: "e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:59.006734 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.006701    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:42:59.006734 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.006732    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wd9\" (UniqueName: \"kubernetes.io/projected/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4-kube-api-access-98wd9\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:42:59.285761 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.285731    2553 generic.go:356] "Generic (PLEG): container finished" podID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerID="655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba" exitCode=0
Sep 05 02:42:59.285929 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.285777    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"
Sep 05 02:42:59.285929 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.285815    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" event={"ID":"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4","Type":"ContainerDied","Data":"655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba"}
Sep 05 02:42:59.285929 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.285868    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl" event={"ID":"e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4","Type":"ContainerDied","Data":"a3162c1bdb634c8227eefd6fac0c71e84f1340adb65e266009a586c950974c58"}
Sep 05 02:42:59.285929 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.285893    2553 scope.go:117] "RemoveContainer" containerID="655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba"
Sep 05 02:42:59.295437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.295419    2553 scope.go:117] "RemoveContainer" containerID="655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba"
Sep 05 02:42:59.295704 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:42:59.295688    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba\": container with ID starting with 655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba not found: ID does not exist" containerID="655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba"
Sep 05 02:42:59.295744 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.295713    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba"} err="failed to get container status \"655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba\": rpc error: code = NotFound desc = could not find container \"655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba\": container with ID starting with 655d1f0b8f91b8ccf465033bd543f102259db5d7f5d0bb8794f13d2f98fd04ba not found: ID does not exist"
Sep 05 02:42:59.307703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.307682    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"]
Sep 05 02:42:59.315477 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:42:59.315455    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9987cl"]
Sep 05 02:43:00.394083 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:00.394053    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" path="/var/lib/kubelet/pods/e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4/volumes"
Sep 05 02:43:01.009733 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.009694    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"]
Sep 05 02:43:01.010257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.010233    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main"
Sep 05 02:43:01.010257 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.010260    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main"
Sep 05 02:43:01.010428 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.010309    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main"
Sep 05 02:43:01.010428 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.010318    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main"
Sep 05 02:43:01.010428 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.010416    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f5d2ab-9509-4cfa-a0bd-3abeccf29ff4" containerName="main"
Sep 05 02:43:01.010530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.010430    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4120da62-15ba-422a-9ae3-e4dc5831f03b" containerName="main"
Sep 05 02:43:01.015212 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.015188    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.018915 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.018659    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-rgwqs\""
Sep 05 02:43:01.018915 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.018706    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 05 02:43:01.018915 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.018731    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 05 02:43:01.018915 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.018753    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 05 02:43:01.019326 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.019029    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-kgkzx\""
Sep 05 02:43:01.022737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.022712    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"]
Sep 05 02:43:01.121686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.121662    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e94621cd-b983-4f23-aca0-0325e4c705e9-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.121830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.121705    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4564d\" (UniqueName: \"kubernetes.io/projected/e94621cd-b983-4f23-aca0-0325e4c705e9-kube-api-access-4564d\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.222276 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.222249    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e94621cd-b983-4f23-aca0-0325e4c705e9-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.222393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.222282    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4564d\" (UniqueName: \"kubernetes.io/projected/e94621cd-b983-4f23-aca0-0325e4c705e9-kube-api-access-4564d\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.224697 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.224677    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e94621cd-b983-4f23-aca0-0325e4c705e9-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.230150 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.230132    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4564d\" (UniqueName: \"kubernetes.io/projected/e94621cd-b983-4f23-aca0-0325e4c705e9-kube-api-access-4564d\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.327608 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.327581    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:01.598403 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:01.598377    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"]
Sep 05 02:43:01.600647 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:43:01.600619    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94621cd_b983_4f23_aca0_0325e4c705e9.slice/crio-13ec2e7af457a46f59368567998f3cb14a5fa083cf979623b08077780251e332 WatchSource:0}: Error finding container 13ec2e7af457a46f59368567998f3cb14a5fa083cf979623b08077780251e332: Status 404 returned error can't find the container with id 13ec2e7af457a46f59368567998f3cb14a5fa083cf979623b08077780251e332
Sep 05 02:43:02.302961 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:02.302929    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" event={"ID":"e94621cd-b983-4f23-aca0-0325e4c705e9","Type":"ContainerStarted","Data":"5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea"}
Sep 05 02:43:02.303146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:02.302989    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" event={"ID":"e94621cd-b983-4f23-aca0-0325e4c705e9","Type":"ContainerStarted","Data":"13ec2e7af457a46f59368567998f3cb14a5fa083cf979623b08077780251e332"}
Sep 05 02:43:02.303146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:02.303021    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:43:02.321478 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:02.321430    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podStartSLOduration=2.321415982 podStartE2EDuration="2.321415982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:02.319175222 +0000 UTC m=+1492.518538239" watchObservedRunningTime="2026-09-05 02:43:02.321415982 +0000 UTC m=+1492.520778999"
Sep 05 02:43:33.309250 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:43:33.309167    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:45:10.243789 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:10.243705    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"]
Sep 05 02:45:13.309532 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:13.309489    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:23.309753 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:23.309716    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:25.264300 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:25.264262    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main" containerID="cri-o://5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea" gracePeriod=45
Sep 05 02:45:33.309778 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:33.309740    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:33.310284 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:33.309863    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:45:43.309983 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:43.309946    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:53.310264 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:53.310219    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:45:55.510367 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.510344    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:45:55.666700 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.666674    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e94621cd-b983-4f23-aca0-0325e4c705e9-tls-certs\") pod \"e94621cd-b983-4f23-aca0-0325e4c705e9\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") "
Sep 05 02:45:55.666864 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.666729    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4564d\" (UniqueName: \"kubernetes.io/projected/e94621cd-b983-4f23-aca0-0325e4c705e9-kube-api-access-4564d\") pod \"e94621cd-b983-4f23-aca0-0325e4c705e9\" (UID: \"e94621cd-b983-4f23-aca0-0325e4c705e9\") "
Sep 05 02:45:55.668781 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.668756    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94621cd-b983-4f23-aca0-0325e4c705e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e94621cd-b983-4f23-aca0-0325e4c705e9" (UID: "e94621cd-b983-4f23-aca0-0325e4c705e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:55.668892 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.668855    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94621cd-b983-4f23-aca0-0325e4c705e9-kube-api-access-4564d" (OuterVolumeSpecName: "kube-api-access-4564d") pod "e94621cd-b983-4f23-aca0-0325e4c705e9" (UID: "e94621cd-b983-4f23-aca0-0325e4c705e9"). InnerVolumeSpecName "kube-api-access-4564d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:55.768217 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.768191    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e94621cd-b983-4f23-aca0-0325e4c705e9-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:45:55.768217 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:55.768215    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4564d\" (UniqueName: \"kubernetes.io/projected/e94621cd-b983-4f23-aca0-0325e4c705e9-kube-api-access-4564d\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:45:56.117779 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.117738    2553 generic.go:356] "Generic (PLEG): container finished" podID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerID="5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea" exitCode=0
Sep 05 02:45:56.117950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.117800    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"
Sep 05 02:45:56.117950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.117829    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" event={"ID":"e94621cd-b983-4f23-aca0-0325e4c705e9","Type":"ContainerDied","Data":"5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea"}
Sep 05 02:45:56.117950 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.117921    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q" event={"ID":"e94621cd-b983-4f23-aca0-0325e4c705e9","Type":"ContainerDied","Data":"13ec2e7af457a46f59368567998f3cb14a5fa083cf979623b08077780251e332"}
Sep 05 02:45:56.118055 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.117952    2553 scope.go:117] "RemoveContainer" containerID="5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea"
Sep 05 02:45:56.127149 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.127126    2553 scope.go:117] "RemoveContainer" containerID="5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea"
Sep 05 02:45:56.127424 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:45:56.127405    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea\": container with ID starting with 5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea not found: ID does not exist" containerID="5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea"
Sep 05 02:45:56.127496 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.127430    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea"} err="failed to get container status \"5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea\": rpc error: code = NotFound desc = could not find container \"5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea\": container with ID starting with 5886d7a68e975d660144c932ed123bfd399eaeb9d2cb125e2b8f52477a0ba3ea not found: ID does not exist"
Sep 05 02:45:56.140531 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.140506    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"]
Sep 05 02:45:56.148155 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.148135    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5vg7q"]
Sep 05 02:45:56.393435 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:45:56.393357    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" path="/var/lib/kubelet/pods/e94621cd-b983-4f23-aca0-0325e4c705e9/volumes"
Sep 05 02:46:04.505443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.505407    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"]
Sep 05 02:46:04.505982 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.505953    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main"
Sep 05 02:46:04.506093 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.505986    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main"
Sep 05 02:46:04.506151 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.506108    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94621cd-b983-4f23-aca0-0325e4c705e9" containerName="main"
Sep 05 02:46:04.510396 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.510375    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.513833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.513792    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 05 02:46:04.513833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.513822    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 05 02:46:04.513992 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.513803    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 05 02:46:04.513992 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.513843    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-xllc5\""
Sep 05 02:46:04.522048 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.522026    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"]
Sep 05 02:46:04.641524 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.641497    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-model-cache\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.641524 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.641527    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.641707 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.641594    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.641707 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.641647    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-home\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.641707 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.641665    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqht7\" (UniqueName: \"kubernetes.io/projected/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-kube-api-access-lqht7\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.641707 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.641686    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-dshm\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.742966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.742937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-model-cache\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.742966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.742972    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743031    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743062    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-home\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743088    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqht7\" (UniqueName: \"kubernetes.io/projected/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-kube-api-access-lqht7\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743118    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-dshm\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743395 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743372    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-model-cache\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743459 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743420    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.743718 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.743689    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-home\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.745258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.745232    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-dshm\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.745533 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.745512    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.750479 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.750454    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqht7\" (UniqueName: \"kubernetes.io/projected/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-kube-api-access-lqht7\") pod \"canary-v1-kserve-65d58ccc9-9b9xs\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:04.820472 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:04.820444    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:05.099655 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:05.099511    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"]
Sep 05 02:46:05.104612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:05.104588    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:46:05.163140 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:05.163105    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" event={"ID":"0a185ebb-7fb0-4c5f-840e-44b6073f1e72","Type":"ContainerStarted","Data":"aaafb4721cbd06ff0cbd33557e56175c11219634eaf94acb0a2b0f3ce5c09efc"}
Sep 05 02:46:07.176331 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:07.176292    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" event={"ID":"0a185ebb-7fb0-4c5f-840e-44b6073f1e72","Type":"ContainerStarted","Data":"508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8"}
Sep 05 02:46:07.198873 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:07.198825    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" podStartSLOduration=2.065348121 podStartE2EDuration="3.19881121s" totalImagesPullingTime="1.133463089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:07.197962548 +0000 UTC m=+1677.397325545" watchObservedRunningTime="2026-09-05 02:46:07.19881121 +0000 UTC m=+1677.398174206"
Sep 05 02:46:07.821319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:07.821286    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:14.820600 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:14.820553    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:14.833729 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:14.833699    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:15.226472 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:15.226385    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:46:35.635812 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.635739    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"]
Sep 05 02:46:35.639185 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.639156    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.642746 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.642710    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-tws9t\""
Sep 05 02:46:35.642746 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.642745    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 05 02:46:35.643011 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.642773    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 05 02:46:35.643011 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.642745    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 05 02:46:35.655099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.655076    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"]
Sep 05 02:46:35.704517 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.704489    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjn9\" (UniqueName: \"kubernetes.io/projected/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-kube-api-access-xrjn9\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.704653 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.704529    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.704653 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.704628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.704737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.704661    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.704737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.704698    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.704737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.704723    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.805728 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.805703    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjn9\" (UniqueName: \"kubernetes.io/projected/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-kube-api-access-xrjn9\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.805857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.805740    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.805857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.805770    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.805857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.805785    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.806004 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.805912    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.806004 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.805964    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.806154 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.806134    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.806219 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.806159    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.806219 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.806202    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.807993 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.807969    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.808295 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.808277    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.813177 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.813157    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjn9\" (UniqueName: \"kubernetes.io/projected/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-kube-api-access-xrjn9\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:35.863217 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.863188    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"]
Sep 05 02:46:35.866529 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.866514    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:35.869022 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.869002    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-854tb\""
Sep 05 02:46:35.877306 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.877286    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"]
Sep 05 02:46:35.950226 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:35.950150    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:36.008392 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.008259    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb4a31-5962-4175-94e8-35e064af1fe9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.008392 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.008317    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqh8\" (UniqueName: \"kubernetes.io/projected/f3eb4a31-5962-4175-94e8-35e064af1fe9-kube-api-access-4wqh8\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.109441 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.109411    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb4a31-5962-4175-94e8-35e064af1fe9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.109615 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.109468    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqh8\" (UniqueName: \"kubernetes.io/projected/f3eb4a31-5962-4175-94e8-35e064af1fe9-kube-api-access-4wqh8\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.111880 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.111862    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb4a31-5962-4175-94e8-35e064af1fe9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.118555 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.118516    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqh8\" (UniqueName: \"kubernetes.io/projected/f3eb4a31-5962-4175-94e8-35e064af1fe9-kube-api-access-4wqh8\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.179696 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.179669    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:36.221825 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.221793    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"]
Sep 05 02:46:36.224530 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:46:36.224504    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee16cf74_0fb1_4e8f_81b6_fb4dc87301c1.slice/crio-69e9af812806d038835c7d52273857fd85a9a9cc7600f120104dd77976ca7f7d WatchSource:0}: Error finding container 69e9af812806d038835c7d52273857fd85a9a9cc7600f120104dd77976ca7f7d: Status 404 returned error can't find the container with id 69e9af812806d038835c7d52273857fd85a9a9cc7600f120104dd77976ca7f7d
Sep 05 02:46:36.319611 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.319586    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" event={"ID":"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1","Type":"ContainerStarted","Data":"364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"}
Sep 05 02:46:36.319728 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.319638    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" event={"ID":"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1","Type":"ContainerStarted","Data":"69e9af812806d038835c7d52273857fd85a9a9cc7600f120104dd77976ca7f7d"}
Sep 05 02:46:36.337642 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.337592    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" podStartSLOduration=1.3375776959999999 podStartE2EDuration="1.337577696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:36.336437218 +0000 UTC m=+1706.535800214" watchObservedRunningTime="2026-09-05 02:46:36.337577696 +0000 UTC m=+1706.536940686"
Sep 05 02:46:36.453322 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.453297    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"]
Sep 05 02:46:36.455170 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:46:36.455142    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3eb4a31_5962_4175_94e8_35e064af1fe9.slice/crio-feff3de4440afcbd08265b805876c99d6990c9d5c020c5eb5630c3762506a35b WatchSource:0}: Error finding container feff3de4440afcbd08265b805876c99d6990c9d5c020c5eb5630c3762506a35b: Status 404 returned error can't find the container with id feff3de4440afcbd08265b805876c99d6990c9d5c020c5eb5630c3762506a35b
Sep 05 02:46:36.951020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:36.950986    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:37.326484 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:37.326448    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" event={"ID":"f3eb4a31-5962-4175-94e8-35e064af1fe9","Type":"ContainerStarted","Data":"c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87"}
Sep 05 02:46:37.326706 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:37.326511    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" event={"ID":"f3eb4a31-5962-4175-94e8-35e064af1fe9","Type":"ContainerStarted","Data":"feff3de4440afcbd08265b805876c99d6990c9d5c020c5eb5630c3762506a35b"}
Sep 05 02:46:37.326813 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:37.326792    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:46:37.344611 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:37.344556    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podStartSLOduration=2.344543084 podStartE2EDuration="2.344543084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:37.342410816 +0000 UTC m=+1707.541773809" watchObservedRunningTime="2026-09-05 02:46:37.344543084 +0000 UTC m=+1707.543906318"
Sep 05 02:46:45.950946 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:45.950911    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:45.963348 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:45.963315    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:46:46.386157 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:46:46.386129    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:47:09.338242 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:47:09.338210    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:47:58.198445 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:47:58.198405    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"]
Sep 05 02:48:13.218790 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.218721    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" podUID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" containerName="main" containerID="cri-o://508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8" gracePeriod=45
Sep 05 02:48:13.222975 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.222942    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" podUID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": EOF"
Sep 05 02:48:13.463245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.463222    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:48:13.527808 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.527744    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqht7\" (UniqueName: \"kubernetes.io/projected/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-kube-api-access-lqht7\") pod \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") "
Sep 05 02:48:13.527808 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.527783    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tmp-dir\") pod \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") "
Sep 05 02:48:13.528008 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.527818    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-dshm\") pod \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") "
Sep 05 02:48:13.528008 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.527853    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tls-certs\") pod \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") "
Sep 05 02:48:13.528008 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.527891    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-model-cache\") pod \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") "
Sep 05 02:48:13.528008 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.527912    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-home\") pod \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\" (UID: \"0a185ebb-7fb0-4c5f-840e-44b6073f1e72\") "
Sep 05 02:48:13.528223 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.528071    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a185ebb-7fb0-4c5f-840e-44b6073f1e72" (UID: "0a185ebb-7fb0-4c5f-840e-44b6073f1e72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.528223 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.528171    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a185ebb-7fb0-4c5f-840e-44b6073f1e72" (UID: "0a185ebb-7fb0-4c5f-840e-44b6073f1e72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.528223 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.528184    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-home" (OuterVolumeSpecName: "home") pod "0a185ebb-7fb0-4c5f-840e-44b6073f1e72" (UID: "0a185ebb-7fb0-4c5f-840e-44b6073f1e72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.528341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.528297    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.528341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.528317    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.528341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.528330    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.530020 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.529990    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-dshm" (OuterVolumeSpecName: "dshm") pod "0a185ebb-7fb0-4c5f-840e-44b6073f1e72" (UID: "0a185ebb-7fb0-4c5f-840e-44b6073f1e72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.530118 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.530047    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-kube-api-access-lqht7" (OuterVolumeSpecName: "kube-api-access-lqht7") pod "0a185ebb-7fb0-4c5f-840e-44b6073f1e72" (UID: "0a185ebb-7fb0-4c5f-840e-44b6073f1e72"). InnerVolumeSpecName "kube-api-access-lqht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:13.530118 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.530078    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a185ebb-7fb0-4c5f-840e-44b6073f1e72" (UID: "0a185ebb-7fb0-4c5f-840e-44b6073f1e72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:13.628892 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.628868    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqht7\" (UniqueName: \"kubernetes.io/projected/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-kube-api-access-lqht7\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.628892 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.628891    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.629026 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.628900    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a185ebb-7fb0-4c5f-840e-44b6073f1e72-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.786518 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.786485    2553 generic.go:356] "Generic (PLEG): container finished" podID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" containerID="508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8" exitCode=0
Sep 05 02:48:13.786686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.786529    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"
Sep 05 02:48:13.786686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.786530    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" event={"ID":"0a185ebb-7fb0-4c5f-840e-44b6073f1e72","Type":"ContainerDied","Data":"508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8"}
Sep 05 02:48:13.786686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.786659    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs" event={"ID":"0a185ebb-7fb0-4c5f-840e-44b6073f1e72","Type":"ContainerDied","Data":"aaafb4721cbd06ff0cbd33557e56175c11219634eaf94acb0a2b0f3ce5c09efc"}
Sep 05 02:48:13.786686 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.786683    2553 scope.go:117] "RemoveContainer" containerID="508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8"
Sep 05 02:48:13.795976 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.795959    2553 scope.go:117] "RemoveContainer" containerID="508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8"
Sep 05 02:48:13.796221 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:48:13.796203    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8\": container with ID starting with 508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8 not found: ID does not exist" containerID="508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8"
Sep 05 02:48:13.796290 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.796229    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8"} err="failed to get container status \"508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8\": rpc error: code = NotFound desc = could not find container \"508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8\": container with ID starting with 508ce17cb506a42ece3a9d7f0201cc154af3e0d4c9edf6c76850e98cc903bda8 not found: ID does not exist"
Sep 05 02:48:13.807530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.807503    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"]
Sep 05 02:48:13.810943 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:13.810921    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9b9xs"]
Sep 05 02:48:14.393895 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:14.393863    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" path="/var/lib/kubelet/pods/0a185ebb-7fb0-4c5f-840e-44b6073f1e72/volumes"
Sep 05 02:48:34.713885 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:34.713844    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"]
Sep 05 02:48:34.718412 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:34.718384    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"]
Sep 05 02:48:34.724844 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:48:34.724810    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:48:34.724844 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:48:34.724844 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:48:34.724844 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315" cmd=["/bin/sleep","15"]
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:34.724858    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" message="" err=<
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:48:34.724870    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" podUID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" containerName="main" containerID="cri-o://364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"
Sep 05 02:48:34.725146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:34.724902    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" podUID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" containerName="main" containerID="cri-o://364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315" gracePeriod=60
Sep 05 02:48:35.072584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.072538    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:48:35.211675 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211643    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-dshm\") pod \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") "
Sep 05 02:48:35.211843 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211691    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-model-cache\") pod \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") "
Sep 05 02:48:35.211843 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211764    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tmp-dir\") pod \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") "
Sep 05 02:48:35.211843 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211786    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-home\") pod \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") "
Sep 05 02:48:35.211843 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211821    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrjn9\" (UniqueName: \"kubernetes.io/projected/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-kube-api-access-xrjn9\") pod \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") "
Sep 05 02:48:35.211843 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211841    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tls-certs\") pod \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\" (UID: \"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1\") "
Sep 05 02:48:35.212168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.211988    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" (UID: "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.212168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.212013    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" (UID: "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.212168 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.212064    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-home" (OuterVolumeSpecName: "home") pod "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" (UID: "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.212280 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.212172    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.212280 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.212189    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.212280 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.212201    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.213929 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.213909    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-dshm" (OuterVolumeSpecName: "dshm") pod "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" (UID: "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.214050 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.214033    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" (UID: "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:35.214084 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.214049    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-kube-api-access-xrjn9" (OuterVolumeSpecName: "kube-api-access-xrjn9") pod "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" (UID: "ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1"). InnerVolumeSpecName "kube-api-access-xrjn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:35.312599 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.312574    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjn9\" (UniqueName: \"kubernetes.io/projected/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-kube-api-access-xrjn9\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.312599 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.312595    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.312764 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.312605    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.895703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.895671    2553 generic.go:356] "Generic (PLEG): container finished" podID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" containerID="364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315" exitCode=0
Sep 05 02:48:35.896103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.895717    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"
Sep 05 02:48:35.896103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.895760    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" event={"ID":"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1","Type":"ContainerDied","Data":"364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"}
Sep 05 02:48:35.896103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.895810    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2" event={"ID":"ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1","Type":"ContainerDied","Data":"69e9af812806d038835c7d52273857fd85a9a9cc7600f120104dd77976ca7f7d"}
Sep 05 02:48:35.896103 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.895834    2553 scope.go:117] "RemoveContainer" containerID="364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"
Sep 05 02:48:35.905152 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.905135    2553 scope.go:117] "RemoveContainer" containerID="364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"
Sep 05 02:48:35.905413 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:48:35.905393    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315\": container with ID starting with 364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315 not found: ID does not exist" containerID="364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"
Sep 05 02:48:35.905458 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.905421    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315"} err="failed to get container status \"364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315\": rpc error: code = NotFound desc = could not find container \"364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315\": container with ID starting with 364b80d663bc7d8e8d5a3f1d82e856c14ff17f74f8561e15d4b4f9575e74b315 not found: ID does not exist"
Sep 05 02:48:35.917596 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.917555    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"]
Sep 05 02:48:35.921472 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:35.921450    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ncst2"]
Sep 05 02:48:36.392992 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:36.392952    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" path="/var/lib/kubelet/pods/ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1/volumes"
Sep 05 02:48:39.338466 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:39.338430    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:48:49.338608 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:49.338548    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:48:49.738139 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:49.738053    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main" containerID="cri-o://c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87" gracePeriod=45
Sep 05 02:48:59.339047 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:59.339003    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:48:59.339488 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:48:59.339102    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:49:09.338706 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:09.338659    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:49:19.338790 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:19.338747    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:49:19.971532 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:19.971509    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:49:20.079131 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.079095    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb4a31-5962-4175-94e8-35e064af1fe9-tls-certs\") pod \"f3eb4a31-5962-4175-94e8-35e064af1fe9\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") "
Sep 05 02:49:20.079289 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.079161    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wqh8\" (UniqueName: \"kubernetes.io/projected/f3eb4a31-5962-4175-94e8-35e064af1fe9-kube-api-access-4wqh8\") pod \"f3eb4a31-5962-4175-94e8-35e064af1fe9\" (UID: \"f3eb4a31-5962-4175-94e8-35e064af1fe9\") "
Sep 05 02:49:20.081212 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.081178    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3eb4a31-5962-4175-94e8-35e064af1fe9-kube-api-access-4wqh8" (OuterVolumeSpecName: "kube-api-access-4wqh8") pod "f3eb4a31-5962-4175-94e8-35e064af1fe9" (UID: "f3eb4a31-5962-4175-94e8-35e064af1fe9"). InnerVolumeSpecName "kube-api-access-4wqh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:49:20.081321 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.081190    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3eb4a31-5962-4175-94e8-35e064af1fe9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3eb4a31-5962-4175-94e8-35e064af1fe9" (UID: "f3eb4a31-5962-4175-94e8-35e064af1fe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:49:20.109459 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.109433    2553 generic.go:356] "Generic (PLEG): container finished" podID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerID="c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87" exitCode=0
Sep 05 02:49:20.109591 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.109478    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"
Sep 05 02:49:20.109591 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.109518    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" event={"ID":"f3eb4a31-5962-4175-94e8-35e064af1fe9","Type":"ContainerDied","Data":"c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87"}
Sep 05 02:49:20.109677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.109590    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26" event={"ID":"f3eb4a31-5962-4175-94e8-35e064af1fe9","Type":"ContainerDied","Data":"feff3de4440afcbd08265b805876c99d6990c9d5c020c5eb5630c3762506a35b"}
Sep 05 02:49:20.109677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.109621    2553 scope.go:117] "RemoveContainer" containerID="c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87"
Sep 05 02:49:20.118501 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.118479    2553 scope.go:117] "RemoveContainer" containerID="c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87"
Sep 05 02:49:20.118792 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:49:20.118771    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87\": container with ID starting with c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87 not found: ID does not exist" containerID="c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87"
Sep 05 02:49:20.118861 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.118802    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87"} err="failed to get container status \"c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87\": rpc error: code = NotFound desc = could not find container \"c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87\": container with ID starting with c219888b5cf61c6d38d0c3211870216f11490ce2a519ece608ffc7165b5dcb87 not found: ID does not exist"
Sep 05 02:49:20.139382 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.139356    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"]
Sep 05 02:49:20.143746 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.143726    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-759pg26"]
Sep 05 02:49:20.186735 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.186709    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb4a31-5962-4175-94e8-35e064af1fe9-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:49:20.186827 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.186739    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wqh8\" (UniqueName: \"kubernetes.io/projected/f3eb4a31-5962-4175-94e8-35e064af1fe9-kube-api-access-4wqh8\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:49:20.393034 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:20.392953    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" path="/var/lib/kubelet/pods/f3eb4a31-5962-4175-94e8-35e064af1fe9/volumes"
Sep 05 02:49:34.838276 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838189    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"]
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838552    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838579    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838593    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838598    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838615    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838621    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838681    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a185ebb-7fb0-4c5f-840e-44b6073f1e72" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838689    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3eb4a31-5962-4175-94e8-35e064af1fe9" containerName="main"
Sep 05 02:49:34.838865 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.838700    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee16cf74-0fb1-4e8f-81b6-fb4dc87301c1" containerName="main"
Sep 05 02:49:34.843282 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.843261    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:34.847204 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.847179    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 05 02:49:34.847319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.847208    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 05 02:49:34.847319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.847241    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 05 02:49:34.847319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.847302    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-brvv8\""
Sep 05 02:49:34.847319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.847312    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-2mz9j\""
Sep 05 02:49:34.854086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:34.854064    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"]
Sep 05 02:49:35.000921 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.000893    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.001086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.000930    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.001086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.000957    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.001086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.000983    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eff8504-8237-4b44-a640-e86f64445c32-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.001086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.001018    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtcp\" (UniqueName: \"kubernetes.io/projected/0eff8504-8237-4b44-a640-e86f64445c32-kube-api-access-jqtcp\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.001086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.001048    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101464    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101492    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101513    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101775 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101532    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eff8504-8237-4b44-a640-e86f64445c32-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101775 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101615    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtcp\" (UniqueName: \"kubernetes.io/projected/0eff8504-8237-4b44-a640-e86f64445c32-kube-api-access-jqtcp\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101775 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101657    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.101987 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.101965    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.102040 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.102008    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.102040 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.102020    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.103995 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.103976    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.104102 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.104086    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eff8504-8237-4b44-a640-e86f64445c32-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.109822 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.109802    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtcp\" (UniqueName: \"kubernetes.io/projected/0eff8504-8237-4b44-a640-e86f64445c32-kube-api-access-jqtcp\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.153338 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.153314    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:35.184166 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.184136    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"]
Sep 05 02:49:35.188438 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.188417    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.191086 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.191068    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-fvfnr\""
Sep 05 02:49:35.199043 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.198950    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"]
Sep 05 02:49:35.303214 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.303176    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvqh\" (UniqueName: \"kubernetes.io/projected/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-kube-api-access-xkvqh\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.303377 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.303338    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.404580 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.404476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.404580 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.404525    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvqh\" (UniqueName: \"kubernetes.io/projected/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-kube-api-access-xkvqh\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.406870 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.406847    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.411830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.411804    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvqh\" (UniqueName: \"kubernetes.io/projected/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-kube-api-access-xkvqh\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.433819 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.433800    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"]
Sep 05 02:49:35.436605 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:49:35.436580    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff8504_8237_4b44_a640_e86f64445c32.slice/crio-f5b691756830ce94538130792babbd35693808d479d422bbb3317e9e43b7cd8d WatchSource:0}: Error finding container f5b691756830ce94538130792babbd35693808d479d422bbb3317e9e43b7cd8d: Status 404 returned error can't find the container with id f5b691756830ce94538130792babbd35693808d479d422bbb3317e9e43b7cd8d
Sep 05 02:49:35.503535 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.503513    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:35.773148 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:35.773119    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"]
Sep 05 02:49:35.774886 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:49:35.774858    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbc9cbf_028a_4d63_9e21_3c386c9e21d8.slice/crio-1879fe2a1c1f325205a63a04c281609d99bc5688a93066f0fc94f7fc82bad781 WatchSource:0}: Error finding container 1879fe2a1c1f325205a63a04c281609d99bc5688a93066f0fc94f7fc82bad781: Status 404 returned error can't find the container with id 1879fe2a1c1f325205a63a04c281609d99bc5688a93066f0fc94f7fc82bad781
Sep 05 02:49:36.197801 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:36.197711    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" event={"ID":"0eff8504-8237-4b44-a640-e86f64445c32","Type":"ContainerStarted","Data":"f5b691756830ce94538130792babbd35693808d479d422bbb3317e9e43b7cd8d"}
Sep 05 02:49:36.200102 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:36.200063    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" event={"ID":"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8","Type":"ContainerStarted","Data":"e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a"}
Sep 05 02:49:36.200251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:36.200121    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" event={"ID":"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8","Type":"ContainerStarted","Data":"1879fe2a1c1f325205a63a04c281609d99bc5688a93066f0fc94f7fc82bad781"}
Sep 05 02:49:36.200430 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:36.200369    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:49:36.221101 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:36.221048    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" podStartSLOduration=1.221029178 podStartE2EDuration="1.221029178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:49:36.220788566 +0000 UTC m=+1886.420151555" watchObservedRunningTime="2026-09-05 02:49:36.221029178 +0000 UTC m=+1886.420392167"
Sep 05 02:49:37.208662 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:37.208604    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" event={"ID":"0eff8504-8237-4b44-a640-e86f64445c32","Type":"ContainerStarted","Data":"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"}
Sep 05 02:49:37.209108 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:37.208830    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:38.216746 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:38.216707    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" event={"ID":"0eff8504-8237-4b44-a640-e86f64445c32","Type":"ContainerStarted","Data":"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"}
Sep 05 02:49:38.237526 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:38.237476    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" podStartSLOduration=2.705776588 podStartE2EDuration="4.23746144s" totalImagesPullingTime="1.531684852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:49:38.235122074 +0000 UTC m=+1888.434485085" watchObservedRunningTime="2026-09-05 02:49:38.23746144 +0000 UTC m=+1888.436824432"
Sep 05 02:49:39.153841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:39.153807    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:45.154250 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:45.154214    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:45.166612 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:45.166581    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:45.261435 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:45.261408    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:49:56.268179 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:49:56.268141    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:50:07.210170 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:07.210138    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:50:40.942128 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:40.942093    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"]
Sep 05 02:50:40.945520 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:40.945492    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"]
Sep 05 02:50:40.955827 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:50:40.955788    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:50:40.955827 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:50:40.955827 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:50:40.955827 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e" cmd=["/bin/sleep","15"]
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:40.955851    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" message="" err=<
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:50:40.955866    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="main" containerID="cri-o://4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"
Sep 05 02:50:40.956116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:40.955926    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="main" containerID="cri-o://4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e" gracePeriod=60
Sep 05 02:50:41.167344 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.167311    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="llm-d-routing-sidecar" containerID="cri-o://97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09" gracePeriod=60
Sep 05 02:50:41.250827 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.250762    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8001/health\": dial tcp 10.132.0.44:8001: connect: connection refused"
Sep 05 02:50:41.420910 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.420888    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:50:41.461998 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.461972    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-tmp-dir\") pod \"0eff8504-8237-4b44-a640-e86f64445c32\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") "
Sep 05 02:50:41.462113 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462044    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-model-cache\") pod \"0eff8504-8237-4b44-a640-e86f64445c32\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") "
Sep 05 02:50:41.462113 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462074    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eff8504-8237-4b44-a640-e86f64445c32-tls-certs\") pod \"0eff8504-8237-4b44-a640-e86f64445c32\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") "
Sep 05 02:50:41.462191 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462128    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-dshm\") pod \"0eff8504-8237-4b44-a640-e86f64445c32\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") "
Sep 05 02:50:41.462191 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462159    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqtcp\" (UniqueName: \"kubernetes.io/projected/0eff8504-8237-4b44-a640-e86f64445c32-kube-api-access-jqtcp\") pod \"0eff8504-8237-4b44-a640-e86f64445c32\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") "
Sep 05 02:50:41.462191 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462170    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0eff8504-8237-4b44-a640-e86f64445c32" (UID: "0eff8504-8237-4b44-a640-e86f64445c32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.462328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462190    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-home\") pod \"0eff8504-8237-4b44-a640-e86f64445c32\" (UID: \"0eff8504-8237-4b44-a640-e86f64445c32\") "
Sep 05 02:50:41.462392 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462322    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-model-cache" (OuterVolumeSpecName: "model-cache") pod "0eff8504-8237-4b44-a640-e86f64445c32" (UID: "0eff8504-8237-4b44-a640-e86f64445c32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.462498 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462480    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.462571 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462503    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.462571 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.462535    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-home" (OuterVolumeSpecName: "home") pod "0eff8504-8237-4b44-a640-e86f64445c32" (UID: "0eff8504-8237-4b44-a640-e86f64445c32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.470604 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.468961    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eff8504-8237-4b44-a640-e86f64445c32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eff8504-8237-4b44-a640-e86f64445c32" (UID: "0eff8504-8237-4b44-a640-e86f64445c32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:41.473723 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.473625    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-dshm" (OuterVolumeSpecName: "dshm") pod "0eff8504-8237-4b44-a640-e86f64445c32" (UID: "0eff8504-8237-4b44-a640-e86f64445c32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.473850 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.473821    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eff8504-8237-4b44-a640-e86f64445c32-kube-api-access-jqtcp" (OuterVolumeSpecName: "kube-api-access-jqtcp") pod "0eff8504-8237-4b44-a640-e86f64445c32" (UID: "0eff8504-8237-4b44-a640-e86f64445c32"). InnerVolumeSpecName "kube-api-access-jqtcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:41.525240 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525180    2553 generic.go:356] "Generic (PLEG): container finished" podID="0eff8504-8237-4b44-a640-e86f64445c32" containerID="4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e" exitCode=0
Sep 05 02:50:41.525240 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525204    2553 generic.go:356] "Generic (PLEG): container finished" podID="0eff8504-8237-4b44-a640-e86f64445c32" containerID="97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09" exitCode=0
Sep 05 02:50:41.525240 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525236    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"
Sep 05 02:50:41.525393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" event={"ID":"0eff8504-8237-4b44-a640-e86f64445c32","Type":"ContainerDied","Data":"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"}
Sep 05 02:50:41.525393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525337    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" event={"ID":"0eff8504-8237-4b44-a640-e86f64445c32","Type":"ContainerDied","Data":"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"}
Sep 05 02:50:41.525393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525355    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr" event={"ID":"0eff8504-8237-4b44-a640-e86f64445c32","Type":"ContainerDied","Data":"f5b691756830ce94538130792babbd35693808d479d422bbb3317e9e43b7cd8d"}
Sep 05 02:50:41.525393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.525375    2553 scope.go:117] "RemoveContainer" containerID="4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"
Sep 05 02:50:41.534145 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.534129    2553 scope.go:117] "RemoveContainer" containerID="97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"
Sep 05 02:50:41.541096 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541079    2553 scope.go:117] "RemoveContainer" containerID="4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"
Sep 05 02:50:41.541335 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:50:41.541316    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e\": container with ID starting with 4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e not found: ID does not exist" containerID="4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"
Sep 05 02:50:41.541391 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541345    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"} err="failed to get container status \"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e\": rpc error: code = NotFound desc = could not find container \"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e\": container with ID starting with 4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e not found: ID does not exist"
Sep 05 02:50:41.541391 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541368    2553 scope.go:117] "RemoveContainer" containerID="97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"
Sep 05 02:50:41.541587 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:50:41.541556    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09\": container with ID starting with 97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09 not found: ID does not exist" containerID="97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"
Sep 05 02:50:41.541635 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541592    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"} err="failed to get container status \"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09\": rpc error: code = NotFound desc = could not find container \"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09\": container with ID starting with 97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09 not found: ID does not exist"
Sep 05 02:50:41.541635 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541605    2553 scope.go:117] "RemoveContainer" containerID="4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"
Sep 05 02:50:41.541804 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541788    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e"} err="failed to get container status \"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e\": rpc error: code = NotFound desc = could not find container \"4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e\": container with ID starting with 4dafd7e7e3b143e8e8a1df9ce07d236fb00d9321b9d7d439a0dee3cc5b494b3e not found: ID does not exist"
Sep 05 02:50:41.541843 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541805    2553 scope.go:117] "RemoveContainer" containerID="97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"
Sep 05 02:50:41.542004 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.541983    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09"} err="failed to get container status \"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09\": rpc error: code = NotFound desc = could not find container \"97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09\": container with ID starting with 97c12c697d1d7c2680922442bddcc1a0f627053713cfa2575972ea6bcea05d09 not found: ID does not exist"
Sep 05 02:50:41.563530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.563511    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.563632 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.563534    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtcp\" (UniqueName: \"kubernetes.io/projected/0eff8504-8237-4b44-a640-e86f64445c32-kube-api-access-jqtcp\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.563632 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.563547    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eff8504-8237-4b44-a640-e86f64445c32-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.563632 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.563575    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eff8504-8237-4b44-a640-e86f64445c32-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.571163 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.571141    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"]
Sep 05 02:50:41.574868 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:41.574850    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-5tzfr"]
Sep 05 02:50:42.392721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:42.392689    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eff8504-8237-4b44-a640-e86f64445c32" path="/var/lib/kubelet/pods/0eff8504-8237-4b44-a640-e86f64445c32/volumes"
Sep 05 02:50:47.210327 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:47.210289    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:50:55.965351 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:55.965292    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main" containerID="cri-o://e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a" gracePeriod=45
Sep 05 02:50:57.211005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:50:57.210964    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:51:07.210587 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:07.210477    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:51:07.210976 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:07.210610    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:51:17.210513 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:17.210473    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:51:26.213279 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.213254    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:51:26.331511 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.331483    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-tls-certs\") pod \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") "
Sep 05 02:51:26.331721 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.331610    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkvqh\" (UniqueName: \"kubernetes.io/projected/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-kube-api-access-xkvqh\") pod \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\" (UID: \"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8\") "
Sep 05 02:51:26.333677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.333651    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" (UID: "8cbc9cbf-028a-4d63-9e21-3c386c9e21d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:51:26.333800 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.333739    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-kube-api-access-xkvqh" (OuterVolumeSpecName: "kube-api-access-xkvqh") pod "8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" (UID: "8cbc9cbf-028a-4d63-9e21-3c386c9e21d8"). InnerVolumeSpecName "kube-api-access-xkvqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:51:26.432530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.432504    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvqh\" (UniqueName: \"kubernetes.io/projected/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-kube-api-access-xkvqh\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:51:26.432530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.432530    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:51:26.739681 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.739594    2553 generic.go:356] "Generic (PLEG): container finished" podID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerID="e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a" exitCode=0
Sep 05 02:51:26.739681 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.739643    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"
Sep 05 02:51:26.739905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.739679    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" event={"ID":"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8","Type":"ContainerDied","Data":"e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a"}
Sep 05 02:51:26.739905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.739735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd" event={"ID":"8cbc9cbf-028a-4d63-9e21-3c386c9e21d8","Type":"ContainerDied","Data":"1879fe2a1c1f325205a63a04c281609d99bc5688a93066f0fc94f7fc82bad781"}
Sep 05 02:51:26.739905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.739756    2553 scope.go:117] "RemoveContainer" containerID="e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a"
Sep 05 02:51:26.749906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.749888    2553 scope.go:117] "RemoveContainer" containerID="e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a"
Sep 05 02:51:26.750206 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:51:26.750185    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a\": container with ID starting with e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a not found: ID does not exist" containerID="e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a"
Sep 05 02:51:26.750283 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.750219    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a"} err="failed to get container status \"e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a\": rpc error: code = NotFound desc = could not find container \"e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a\": container with ID starting with e1f016805b777d631376afa1c49455612870bedb13852259abc923d83ec4c44a not found: ID does not exist"
Sep 05 02:51:26.768390 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.768361    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"]
Sep 05 02:51:26.768515 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:26.768417    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched2bdnd"]
Sep 05 02:51:28.392357 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:28.392323    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" path="/var/lib/kubelet/pods/8cbc9cbf-028a-4d63-9e21-3c386c9e21d8/volumes"
Sep 05 02:51:29.300596 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.300544    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"]
Sep 05 02:51:29.301001 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.300982    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main"
Sep 05 02:51:29.301099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301004    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main"
Sep 05 02:51:29.301099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301019    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="main"
Sep 05 02:51:29.301099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301026    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="main"
Sep 05 02:51:29.301099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301052    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="llm-d-routing-sidecar"
Sep 05 02:51:29.301099 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301061    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="llm-d-routing-sidecar"
Sep 05 02:51:29.301350 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301145    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbc9cbf-028a-4d63-9e21-3c386c9e21d8" containerName="main"
Sep 05 02:51:29.301350 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301189    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="llm-d-routing-sidecar"
Sep 05 02:51:29.301350 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.301200    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eff8504-8237-4b44-a640-e86f64445c32" containerName="main"
Sep 05 02:51:29.305699 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.305678    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.308440 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.308416    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 05 02:51:29.309549 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.309527    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 05 02:51:29.309652 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.309549    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 05 02:51:29.309652 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.309594    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-qrchx\""
Sep 05 02:51:29.315072 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.315050    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"]
Sep 05 02:51:29.354342 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.354312    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-model-cache\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.354443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.354353    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.354443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.354381    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.354443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.354430    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-home\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.354590 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.354464    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-dshm\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.354590 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.354492    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nngf\" (UniqueName: \"kubernetes.io/projected/e69894b1-9c2c-4bbe-a8b0-309807c41e45-kube-api-access-2nngf\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455047 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455016    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455053    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-home\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455086    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-dshm\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455133    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nngf\" (UniqueName: \"kubernetes.io/projected/e69894b1-9c2c-4bbe-a8b0-309807c41e45-kube-api-access-2nngf\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455175    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-model-cache\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455418 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455680 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455431    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455680 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455458    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-home\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.455680 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.455643    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-model-cache\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.457185 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.457168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-dshm\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.457473 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.457457    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.463318 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.463298    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nngf\" (UniqueName: \"kubernetes.io/projected/e69894b1-9c2c-4bbe-a8b0-309807c41e45-kube-api-access-2nngf\") pod \"decom-v1-kserve-6cb797fb75-pjck4\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.616905 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.616817    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:29.897804 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:29.897777    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"]
Sep 05 02:51:29.900409 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:51:29.900378    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69894b1_9c2c_4bbe_a8b0_309807c41e45.slice/crio-3e89091f363820980d9471fd3a3c2367c947b3fe1304ce3d9b00e82c0de65e12 WatchSource:0}: Error finding container 3e89091f363820980d9471fd3a3c2367c947b3fe1304ce3d9b00e82c0de65e12: Status 404 returned error can't find the container with id 3e89091f363820980d9471fd3a3c2367c947b3fe1304ce3d9b00e82c0de65e12
Sep 05 02:51:30.761236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:30.761198    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4" event={"ID":"e69894b1-9c2c-4bbe-a8b0-309807c41e45","Type":"ContainerStarted","Data":"34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1"}
Sep 05 02:51:30.761689 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:30.761261    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4" event={"ID":"e69894b1-9c2c-4bbe-a8b0-309807c41e45","Type":"ContainerStarted","Data":"3e89091f363820980d9471fd3a3c2367c947b3fe1304ce3d9b00e82c0de65e12"}
Sep 05 02:51:30.785053 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:30.785005    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4" podStartSLOduration=1.784989479 podStartE2EDuration="1.784989479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:30.783340867 +0000 UTC m=+2000.982703861" watchObservedRunningTime="2026-09-05 02:51:30.784989479 +0000 UTC m=+2000.984352471"
Sep 05 02:51:31.617523 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:31.617492    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:39.618006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:39.617967    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:39.630584 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:39.630538    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:39.815627 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:39.815598    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:51:54.491744 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.491712    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"]
Sep 05 02:51:54.495393 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.495372    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.498016 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.497993    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 05 02:51:54.499308 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.499283    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-9clcd\""
Sep 05 02:51:54.499427 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.499412    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 05 02:51:54.499478 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.499447    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 05 02:51:54.499623 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.499555    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tfjbj\""
Sep 05 02:51:54.504986 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.504966    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"]
Sep 05 02:51:54.557614 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.557581    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2dl\" (UniqueName: \"kubernetes.io/projected/5f0e0251-a94b-455e-9737-9091dc55a91e-kube-api-access-md2dl\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.557746 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.557624    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0e0251-a94b-455e-9737-9091dc55a91e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.658858 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.658834    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2dl\" (UniqueName: \"kubernetes.io/projected/5f0e0251-a94b-455e-9737-9091dc55a91e-kube-api-access-md2dl\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.658956 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.658864    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0e0251-a94b-455e-9737-9091dc55a91e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.661368 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.661344    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0e0251-a94b-455e-9737-9091dc55a91e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.668677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.668657    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2dl\" (UniqueName: \"kubernetes.io/projected/5f0e0251-a94b-455e-9737-9091dc55a91e-kube-api-access-md2dl\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:54.806280 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:54.806248    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:55.078169 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:55.078140    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"]
Sep 05 02:51:55.079933 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:51:55.079904    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0e0251_a94b_455e_9737_9091dc55a91e.slice/crio-e9c988419f5e7a76305dd851511930bd1aa8ed654995160f516735cc8769f795 WatchSource:0}: Error finding container e9c988419f5e7a76305dd851511930bd1aa8ed654995160f516735cc8769f795: Status 404 returned error can't find the container with id e9c988419f5e7a76305dd851511930bd1aa8ed654995160f516735cc8769f795
Sep 05 02:51:55.882260 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:55.882223    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" event={"ID":"5f0e0251-a94b-455e-9737-9091dc55a91e","Type":"ContainerStarted","Data":"9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9"}
Sep 05 02:51:55.882762 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:55.882281    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" event={"ID":"5f0e0251-a94b-455e-9737-9091dc55a91e","Type":"ContainerStarted","Data":"e9c988419f5e7a76305dd851511930bd1aa8ed654995160f516735cc8769f795"}
Sep 05 02:51:55.882762 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:55.882332    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:51:55.902280 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:51:55.902233    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podStartSLOduration=1.902220078 podStartE2EDuration="1.902220078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:55.900217391 +0000 UTC m=+2026.099580386" watchObservedRunningTime="2026-09-05 02:51:55.902220078 +0000 UTC m=+2026.101583071"
Sep 05 02:52:26.888542 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:52:26.888514    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:53:12.793350 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:12.793314    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"]
Sep 05 02:53:27.813849 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:27.813801    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4" podUID="e69894b1-9c2c-4bbe-a8b0-309807c41e45" containerName="main" containerID="cri-o://34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1" gracePeriod=45
Sep 05 02:53:28.065685 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.065629    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:53:28.146979 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.146953    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nngf\" (UniqueName: \"kubernetes.io/projected/e69894b1-9c2c-4bbe-a8b0-309807c41e45-kube-api-access-2nngf\") pod \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") "
Sep 05 02:53:28.147127 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.146997    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tmp-dir\") pod \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") "
Sep 05 02:53:28.147127 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147016    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-dshm\") pod \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") "
Sep 05 02:53:28.147127 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147070    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-model-cache\") pod \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") "
Sep 05 02:53:28.147127 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147093    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tls-certs\") pod \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") "
Sep 05 02:53:28.147127 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147120    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-home\") pod \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\" (UID: \"e69894b1-9c2c-4bbe-a8b0-309807c41e45\") "
Sep 05 02:53:28.147405 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147259    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e69894b1-9c2c-4bbe-a8b0-309807c41e45" (UID: "e69894b1-9c2c-4bbe-a8b0-309807c41e45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:28.147405 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147355    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-model-cache" (OuterVolumeSpecName: "model-cache") pod "e69894b1-9c2c-4bbe-a8b0-309807c41e45" (UID: "e69894b1-9c2c-4bbe-a8b0-309807c41e45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:28.147491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147383    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:53:28.147491 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.147413    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-home" (OuterVolumeSpecName: "home") pod "e69894b1-9c2c-4bbe-a8b0-309807c41e45" (UID: "e69894b1-9c2c-4bbe-a8b0-309807c41e45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:28.149302 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.149268    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69894b1-9c2c-4bbe-a8b0-309807c41e45-kube-api-access-2nngf" (OuterVolumeSpecName: "kube-api-access-2nngf") pod "e69894b1-9c2c-4bbe-a8b0-309807c41e45" (UID: "e69894b1-9c2c-4bbe-a8b0-309807c41e45"). InnerVolumeSpecName "kube-api-access-2nngf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:53:28.149403 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.149301    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e69894b1-9c2c-4bbe-a8b0-309807c41e45" (UID: "e69894b1-9c2c-4bbe-a8b0-309807c41e45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:53:28.149403 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.149306    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-dshm" (OuterVolumeSpecName: "dshm") pod "e69894b1-9c2c-4bbe-a8b0-309807c41e45" (UID: "e69894b1-9c2c-4bbe-a8b0-309807c41e45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:28.247857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.247830    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:53:28.247857 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.247854    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69894b1-9c2c-4bbe-a8b0-309807c41e45-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:53:28.248000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.247865    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:53:28.248000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.247878    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nngf\" (UniqueName: \"kubernetes.io/projected/e69894b1-9c2c-4bbe-a8b0-309807c41e45-kube-api-access-2nngf\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:53:28.248000 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.247891    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69894b1-9c2c-4bbe-a8b0-309807c41e45-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:53:28.324991 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.324918    2553 generic.go:356] "Generic (PLEG): container finished" podID="e69894b1-9c2c-4bbe-a8b0-309807c41e45" containerID="34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1" exitCode=0
Sep 05 02:53:28.324991 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.324968    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"
Sep 05 02:53:28.325189 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.325007    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4" event={"ID":"e69894b1-9c2c-4bbe-a8b0-309807c41e45","Type":"ContainerDied","Data":"34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1"}
Sep 05 02:53:28.325189 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.325064    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4" event={"ID":"e69894b1-9c2c-4bbe-a8b0-309807c41e45","Type":"ContainerDied","Data":"3e89091f363820980d9471fd3a3c2367c947b3fe1304ce3d9b00e82c0de65e12"}
Sep 05 02:53:28.325189 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.325087    2553 scope.go:117] "RemoveContainer" containerID="34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1"
Sep 05 02:53:28.333860 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.333845    2553 scope.go:117] "RemoveContainer" containerID="34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1"
Sep 05 02:53:28.334134 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:53:28.334114    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1\": container with ID starting with 34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1 not found: ID does not exist" containerID="34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1"
Sep 05 02:53:28.334184 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.334143    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1"} err="failed to get container status \"34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1\": rpc error: code = NotFound desc = could not find container \"34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1\": container with ID starting with 34ac681b49e50e550e14e0abe62faf69f493a062eab5996af6fe4258c88ccbe1 not found: ID does not exist"
Sep 05 02:53:28.346423 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.346401    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"]
Sep 05 02:53:28.354949 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.354930    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-pjck4"]
Sep 05 02:53:28.392930 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:28.392906    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69894b1-9c2c-4bbe-a8b0-309807c41e45" path="/var/lib/kubelet/pods/e69894b1-9c2c-4bbe-a8b0-309807c41e45/volumes"
Sep 05 02:53:33.404386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.404354    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"]
Sep 05 02:53:33.404872 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.404753    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69894b1-9c2c-4bbe-a8b0-309807c41e45" containerName="main"
Sep 05 02:53:33.404872 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.404766    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69894b1-9c2c-4bbe-a8b0-309807c41e45" containerName="main"
Sep 05 02:53:33.404872 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.404828    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69894b1-9c2c-4bbe-a8b0-309807c41e45" containerName="main"
Sep 05 02:53:33.409262 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.409244    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.412969 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.412946    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 05 02:53:33.412969 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.412958    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 05 02:53:33.413148 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.412961    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-tgv45\""
Sep 05 02:53:33.413148 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.412998    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 05 02:53:33.417097 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.417071    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"]
Sep 05 02:53:33.489363 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.489338    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tls-certs\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.489363 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.489366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-model-cache\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.489554 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.489384    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgm4\" (UniqueName: \"kubernetes.io/projected/25cbb379-9112-4a10-8dfc-e87b1364eeb9-kube-api-access-8hgm4\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.489554 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.489459    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-home\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.489554 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.489532    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-dshm\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.489690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.489607    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590187 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590160    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-home\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590215    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-dshm\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590241    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tls-certs\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590284    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-model-cache\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590341 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590302    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgm4\" (UniqueName: \"kubernetes.io/projected/25cbb379-9112-4a10-8dfc-e87b1364eeb9-kube-api-access-8hgm4\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590650 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590619    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-home\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590742 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590723    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.590796 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.590760    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-model-cache\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.592468 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.592451    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-dshm\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.592657 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.592640    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tls-certs\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.598787 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.598769    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgm4\" (UniqueName: \"kubernetes.io/projected/25cbb379-9112-4a10-8dfc-e87b1364eeb9-kube-api-access-8hgm4\") pod \"leave-v1-kserve-65684d4f5-t6ghv\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.720740 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.720656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:33.995894 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:33.995863    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"]
Sep 05 02:53:33.998906 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:53:33.998876    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cbb379_9112_4a10_8dfc_e87b1364eeb9.slice/crio-4a3c93565b3eebbb8883521fe9a0c755fe11e6a80bd34d5c0760c04cf1d4bcf5 WatchSource:0}: Error finding container 4a3c93565b3eebbb8883521fe9a0c755fe11e6a80bd34d5c0760c04cf1d4bcf5: Status 404 returned error can't find the container with id 4a3c93565b3eebbb8883521fe9a0c755fe11e6a80bd34d5c0760c04cf1d4bcf5
Sep 05 02:53:34.354208 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:34.354173    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv" event={"ID":"25cbb379-9112-4a10-8dfc-e87b1364eeb9","Type":"ContainerStarted","Data":"eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951"}
Sep 05 02:53:34.354401 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:34.354235    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv" event={"ID":"25cbb379-9112-4a10-8dfc-e87b1364eeb9","Type":"ContainerStarted","Data":"4a3c93565b3eebbb8883521fe9a0c755fe11e6a80bd34d5c0760c04cf1d4bcf5"}
Sep 05 02:53:34.374249 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:34.374202    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv" podStartSLOduration=1.374188422 podStartE2EDuration="1.374188422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:53:34.373857874 +0000 UTC m=+2124.573220867" watchObservedRunningTime="2026-09-05 02:53:34.374188422 +0000 UTC m=+2124.573551415"
Sep 05 02:53:34.721616 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:34.721526    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:41.405623 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.405588    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"]
Sep 05 02:53:41.414744 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.414715    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.417581 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.417544    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 05 02:53:41.421572 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.421502    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"]
Sep 05 02:53:41.467415 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.467389    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-model-cache\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.467534 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.467432    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrgg\" (UniqueName: \"kubernetes.io/projected/c045efd4-d519-48bf-aa84-038c84313b10-kube-api-access-bgrgg\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.467534 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.467511    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-dshm\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.467652 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.467610    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-home\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.467652 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.467639    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c045efd4-d519-48bf-aa84-038c84313b10-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.467729 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.467701    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568421 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568435    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-model-cache\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568468    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrgg\" (UniqueName: \"kubernetes.io/projected/c045efd4-d519-48bf-aa84-038c84313b10-kube-api-access-bgrgg\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-dshm\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568525    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-home\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568545    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c045efd4-d519-48bf-aa84-038c84313b10-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.568937 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568913    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-model-cache\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.569007 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568947    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-home\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.569007 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.568964    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.570924 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.570902    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-dshm\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.571140 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.571124    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c045efd4-d519-48bf-aa84-038c84313b10-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.576025 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.576007    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrgg\" (UniqueName: \"kubernetes.io/projected/c045efd4-d519-48bf-aa84-038c84313b10-kube-api-access-bgrgg\") pod \"leave-v2-kserve-556b4dddf9-btxn6\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.726757 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.726678    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:41.998318 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:41.998263    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"]
Sep 05 02:53:42.000897 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:53:42.000864    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc045efd4_d519_48bf_aa84_038c84313b10.slice/crio-5b112aae87f44a71b48ec6f2b936b823d027395da705a18a0da65cd65336cd12 WatchSource:0}: Error finding container 5b112aae87f44a71b48ec6f2b936b823d027395da705a18a0da65cd65336cd12: Status 404 returned error can't find the container with id 5b112aae87f44a71b48ec6f2b936b823d027395da705a18a0da65cd65336cd12
Sep 05 02:53:42.394973 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:42.394937    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6" event={"ID":"c045efd4-d519-48bf-aa84-038c84313b10","Type":"ContainerStarted","Data":"5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727"}
Sep 05 02:53:42.395137 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:42.394989    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6" event={"ID":"c045efd4-d519-48bf-aa84-038c84313b10","Type":"ContainerStarted","Data":"5b112aae87f44a71b48ec6f2b936b823d027395da705a18a0da65cd65336cd12"}
Sep 05 02:53:42.416600 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:42.416526    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6" podStartSLOduration=1.416511526 podStartE2EDuration="1.416511526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:53:42.414751507 +0000 UTC m=+2132.614114501" watchObservedRunningTime="2026-09-05 02:53:42.416511526 +0000 UTC m=+2132.615874520"
Sep 05 02:53:42.727630 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:42.727523    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:43.721289 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:43.721258    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:43.733908 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:43.733878    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:44.414745 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:44.414716    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:53:51.727481 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:51.727446    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:51.739841 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:51.739813    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:53:52.452906 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:53:52.452872    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:54:05.081820 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:05.081736    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"]
Sep 05 02:54:06.888871 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:06.888830    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:54:11.899161 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:11.899122    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"]
Sep 05 02:54:15.932590 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:15.932541    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"]
Sep 05 02:54:16.888784 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:16.888747    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:54:20.102039 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:20.101999    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main" containerID="cri-o://9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9" gracePeriod=45
Sep 05 02:54:26.889328 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:26.889286    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:54:26.889846 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:26.889392    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:54:26.919536 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:26.919498    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6" podUID="c045efd4-d519-48bf-aa84-038c84313b10" containerName="main" containerID="cri-o://5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727" gracePeriod=45
Sep 05 02:54:27.163597 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.163574    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:54:27.265042 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265015    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgrgg\" (UniqueName: \"kubernetes.io/projected/c045efd4-d519-48bf-aa84-038c84313b10-kube-api-access-bgrgg\") pod \"c045efd4-d519-48bf-aa84-038c84313b10\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") "
Sep 05 02:54:27.265164 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265068    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-model-cache\") pod \"c045efd4-d519-48bf-aa84-038c84313b10\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") "
Sep 05 02:54:27.265164 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265090    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-dshm\") pod \"c045efd4-d519-48bf-aa84-038c84313b10\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") "
Sep 05 02:54:27.265164 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265143    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-home\") pod \"c045efd4-d519-48bf-aa84-038c84313b10\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") "
Sep 05 02:54:27.265302 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265188    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c045efd4-d519-48bf-aa84-038c84313b10-tls-certs\") pod \"c045efd4-d519-48bf-aa84-038c84313b10\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") "
Sep 05 02:54:27.265302 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265245    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-tmp-dir\") pod \"c045efd4-d519-48bf-aa84-038c84313b10\" (UID: \"c045efd4-d519-48bf-aa84-038c84313b10\") "
Sep 05 02:54:27.265441 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265411    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-model-cache" (OuterVolumeSpecName: "model-cache") pod "c045efd4-d519-48bf-aa84-038c84313b10" (UID: "c045efd4-d519-48bf-aa84-038c84313b10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:27.265521 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265456    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-home" (OuterVolumeSpecName: "home") pod "c045efd4-d519-48bf-aa84-038c84313b10" (UID: "c045efd4-d519-48bf-aa84-038c84313b10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:27.265615 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265590    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c045efd4-d519-48bf-aa84-038c84313b10" (UID: "c045efd4-d519-48bf-aa84-038c84313b10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:27.265677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265617    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:27.265677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.265638    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:27.267258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.267230    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c045efd4-d519-48bf-aa84-038c84313b10-kube-api-access-bgrgg" (OuterVolumeSpecName: "kube-api-access-bgrgg") pod "c045efd4-d519-48bf-aa84-038c84313b10" (UID: "c045efd4-d519-48bf-aa84-038c84313b10"). InnerVolumeSpecName "kube-api-access-bgrgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:27.267258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.267231    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-dshm" (OuterVolumeSpecName: "dshm") pod "c045efd4-d519-48bf-aa84-038c84313b10" (UID: "c045efd4-d519-48bf-aa84-038c84313b10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:27.267411 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.267309    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c045efd4-d519-48bf-aa84-038c84313b10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c045efd4-d519-48bf-aa84-038c84313b10" (UID: "c045efd4-d519-48bf-aa84-038c84313b10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:27.366415 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.366380    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c045efd4-d519-48bf-aa84-038c84313b10-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:27.366415 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.366407    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:27.366623 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.366420    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgrgg\" (UniqueName: \"kubernetes.io/projected/c045efd4-d519-48bf-aa84-038c84313b10-kube-api-access-bgrgg\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:27.366623 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.366434    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c045efd4-d519-48bf-aa84-038c84313b10-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:27.613473 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.613440    2553 generic.go:356] "Generic (PLEG): container finished" podID="c045efd4-d519-48bf-aa84-038c84313b10" containerID="5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727" exitCode=0
Sep 05 02:54:27.613676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.613492    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"
Sep 05 02:54:27.613676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.613494    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6" event={"ID":"c045efd4-d519-48bf-aa84-038c84313b10","Type":"ContainerDied","Data":"5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727"}
Sep 05 02:54:27.613676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.613637    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6" event={"ID":"c045efd4-d519-48bf-aa84-038c84313b10","Type":"ContainerDied","Data":"5b112aae87f44a71b48ec6f2b936b823d027395da705a18a0da65cd65336cd12"}
Sep 05 02:54:27.613676 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.613660    2553 scope.go:117] "RemoveContainer" containerID="5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727"
Sep 05 02:54:27.624469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.623687    2553 scope.go:117] "RemoveContainer" containerID="5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727"
Sep 05 02:54:27.624469 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:54:27.624421    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727\": container with ID starting with 5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727 not found: ID does not exist" containerID="5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727"
Sep 05 02:54:27.624469 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.624449    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727"} err="failed to get container status \"5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727\": rpc error: code = NotFound desc = could not find container \"5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727\": container with ID starting with 5becd9ec11c88130dbcd8894723673b1fa0aeb7f6d83fbac357ae246bc0f7727 not found: ID does not exist"
Sep 05 02:54:27.635712 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.635685    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"]
Sep 05 02:54:27.641820 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:27.641797    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-btxn6"]
Sep 05 02:54:28.392738 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:28.392704    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c045efd4-d519-48bf-aa84-038c84313b10" path="/var/lib/kubelet/pods/c045efd4-d519-48bf-aa84-038c84313b10/volumes"
Sep 05 02:54:30.952503 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:30.952465    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv" podUID="25cbb379-9112-4a10-8dfc-e87b1364eeb9" containerName="main" containerID="cri-o://eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951" gracePeriod=45
Sep 05 02:54:31.197615 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.197593    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:54:31.301654 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301622    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-home\") pod \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") "
Sep 05 02:54:31.301826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301661    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hgm4\" (UniqueName: \"kubernetes.io/projected/25cbb379-9112-4a10-8dfc-e87b1364eeb9-kube-api-access-8hgm4\") pod \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") "
Sep 05 02:54:31.301826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301744    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tls-certs\") pod \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") "
Sep 05 02:54:31.301826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301766    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-dshm\") pod \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") "
Sep 05 02:54:31.301826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301785    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tmp-dir\") pod \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") "
Sep 05 02:54:31.301826 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301822    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-model-cache\") pod \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\" (UID: \"25cbb379-9112-4a10-8dfc-e87b1364eeb9\") "
Sep 05 02:54:31.302097 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.301950    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-home" (OuterVolumeSpecName: "home") pod "25cbb379-9112-4a10-8dfc-e87b1364eeb9" (UID: "25cbb379-9112-4a10-8dfc-e87b1364eeb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:31.302160 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.302102    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25cbb379-9112-4a10-8dfc-e87b1364eeb9" (UID: "25cbb379-9112-4a10-8dfc-e87b1364eeb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:31.302160 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.302109    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:31.302236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.302184    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "25cbb379-9112-4a10-8dfc-e87b1364eeb9" (UID: "25cbb379-9112-4a10-8dfc-e87b1364eeb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:31.303855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.303828    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cbb379-9112-4a10-8dfc-e87b1364eeb9-kube-api-access-8hgm4" (OuterVolumeSpecName: "kube-api-access-8hgm4") pod "25cbb379-9112-4a10-8dfc-e87b1364eeb9" (UID: "25cbb379-9112-4a10-8dfc-e87b1364eeb9"). InnerVolumeSpecName "kube-api-access-8hgm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:31.303969 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.303867    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-dshm" (OuterVolumeSpecName: "dshm") pod "25cbb379-9112-4a10-8dfc-e87b1364eeb9" (UID: "25cbb379-9112-4a10-8dfc-e87b1364eeb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:31.303969 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.303878    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25cbb379-9112-4a10-8dfc-e87b1364eeb9" (UID: "25cbb379-9112-4a10-8dfc-e87b1364eeb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:31.402613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.402588    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgm4\" (UniqueName: \"kubernetes.io/projected/25cbb379-9112-4a10-8dfc-e87b1364eeb9-kube-api-access-8hgm4\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:31.402613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.402611    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:31.402776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.402619    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:31.402776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.402627    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:31.402776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.402635    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25cbb379-9112-4a10-8dfc-e87b1364eeb9-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:31.634322 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.634246    2553 generic.go:356] "Generic (PLEG): container finished" podID="25cbb379-9112-4a10-8dfc-e87b1364eeb9" containerID="eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951" exitCode=0
Sep 05 02:54:31.634322 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.634289    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"
Sep 05 02:54:31.634322 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.634300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv" event={"ID":"25cbb379-9112-4a10-8dfc-e87b1364eeb9","Type":"ContainerDied","Data":"eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951"}
Sep 05 02:54:31.634540 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.634341    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv" event={"ID":"25cbb379-9112-4a10-8dfc-e87b1364eeb9","Type":"ContainerDied","Data":"4a3c93565b3eebbb8883521fe9a0c755fe11e6a80bd34d5c0760c04cf1d4bcf5"}
Sep 05 02:54:31.634540 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.634363    2553 scope.go:117] "RemoveContainer" containerID="eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951"
Sep 05 02:54:31.643630 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.643554    2553 scope.go:117] "RemoveContainer" containerID="eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951"
Sep 05 02:54:31.643850 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:54:31.643834    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951\": container with ID starting with eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951 not found: ID does not exist" containerID="eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951"
Sep 05 02:54:31.643902 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.643857    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951"} err="failed to get container status \"eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951\": rpc error: code = NotFound desc = could not find container \"eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951\": container with ID starting with eb0d5157ca088da0eff6f81d3cb028a3a0fd587c6b563fdee4c93321bcebb951 not found: ID does not exist"
Sep 05 02:54:31.656366 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.656341    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"]
Sep 05 02:54:31.662094 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:31.662073    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-t6ghv"]
Sep 05 02:54:32.392658 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:32.392625    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25cbb379-9112-4a10-8dfc-e87b1364eeb9" path="/var/lib/kubelet/pods/25cbb379-9112-4a10-8dfc-e87b1364eeb9/volumes"
Sep 05 02:54:36.888345 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:36.888308    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:54:46.888443 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:46.888397    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:54:50.346085 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.346055    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:54:50.460710 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.460642    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md2dl\" (UniqueName: \"kubernetes.io/projected/5f0e0251-a94b-455e-9737-9091dc55a91e-kube-api-access-md2dl\") pod \"5f0e0251-a94b-455e-9737-9091dc55a91e\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") "
Sep 05 02:54:50.460710 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.460672    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0e0251-a94b-455e-9737-9091dc55a91e-tls-certs\") pod \"5f0e0251-a94b-455e-9737-9091dc55a91e\" (UID: \"5f0e0251-a94b-455e-9737-9091dc55a91e\") "
Sep 05 02:54:50.462830 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.462802    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0e0251-a94b-455e-9737-9091dc55a91e-kube-api-access-md2dl" (OuterVolumeSpecName: "kube-api-access-md2dl") pod "5f0e0251-a94b-455e-9737-9091dc55a91e" (UID: "5f0e0251-a94b-455e-9737-9091dc55a91e"). InnerVolumeSpecName "kube-api-access-md2dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:50.462941 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.462887    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0e0251-a94b-455e-9737-9091dc55a91e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f0e0251-a94b-455e-9737-9091dc55a91e" (UID: "5f0e0251-a94b-455e-9737-9091dc55a91e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:50.562210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.561970    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md2dl\" (UniqueName: \"kubernetes.io/projected/5f0e0251-a94b-455e-9737-9091dc55a91e-kube-api-access-md2dl\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:50.562210 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.562001    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0e0251-a94b-455e-9737-9091dc55a91e-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:54:50.731896 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.731819    2553 generic.go:356] "Generic (PLEG): container finished" podID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerID="9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9" exitCode=0
Sep 05 02:54:50.731896 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.731864    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"
Sep 05 02:54:50.732069 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.731898    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" event={"ID":"5f0e0251-a94b-455e-9737-9091dc55a91e","Type":"ContainerDied","Data":"9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9"}
Sep 05 02:54:50.732069 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.731951    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg" event={"ID":"5f0e0251-a94b-455e-9737-9091dc55a91e","Type":"ContainerDied","Data":"e9c988419f5e7a76305dd851511930bd1aa8ed654995160f516735cc8769f795"}
Sep 05 02:54:50.732069 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.731974    2553 scope.go:117] "RemoveContainer" containerID="9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9"
Sep 05 02:54:50.741354 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.741338    2553 scope.go:117] "RemoveContainer" containerID="9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9"
Sep 05 02:54:50.741618 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:54:50.741597    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9\": container with ID starting with 9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9 not found: ID does not exist" containerID="9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9"
Sep 05 02:54:50.741690 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.741630    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9"} err="failed to get container status \"9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9\": rpc error: code = NotFound desc = could not find container \"9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9\": container with ID starting with 9285a2faf8ae2784290bd8ed494e9bca6876026d5c0e47f46b7ddb94bc4143d9 not found: ID does not exist"
Sep 05 02:54:50.755436 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.755411    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"]
Sep 05 02:54:50.759002 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:50.758982    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69kdhdg"]
Sep 05 02:54:52.392456 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:54:52.392417    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" path="/var/lib/kubelet/pods/5f0e0251-a94b-455e-9737-9091dc55a91e/volumes"
Sep 05 02:55:15.281318 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281280    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"]
Sep 05 02:55:15.281703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281669    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25cbb379-9112-4a10-8dfc-e87b1364eeb9" containerName="main"
Sep 05 02:55:15.281703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281681    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="25cbb379-9112-4a10-8dfc-e87b1364eeb9" containerName="main"
Sep 05 02:55:15.281703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281698    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c045efd4-d519-48bf-aa84-038c84313b10" containerName="main"
Sep 05 02:55:15.281703 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281704    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c045efd4-d519-48bf-aa84-038c84313b10" containerName="main"
Sep 05 02:55:15.281839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281712    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main"
Sep 05 02:55:15.281839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281718    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main"
Sep 05 02:55:15.281839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281781    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c045efd4-d519-48bf-aa84-038c84313b10" containerName="main"
Sep 05 02:55:15.281839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281791    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0e0251-a94b-455e-9737-9091dc55a91e" containerName="main"
Sep 05 02:55:15.281839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.281797    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="25cbb379-9112-4a10-8dfc-e87b1364eeb9" containerName="main"
Sep 05 02:55:15.286512 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.286491    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.289878 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.289855    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-x79pf\""
Sep 05 02:55:15.290005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.289918    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9q65n\""
Sep 05 02:55:15.290005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.289937    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 05 02:55:15.290005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.289937    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 05 02:55:15.290170 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.289941    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 05 02:55:15.293248 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.293228    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"]
Sep 05 02:55:15.357446 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.357415    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstzf\" (UniqueName: \"kubernetes.io/projected/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-kube-api-access-dstzf\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578549szh\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.357633 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.357450    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578549szh\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.458090 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.458057    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstzf\" (UniqueName: \"kubernetes.io/projected/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-kube-api-access-dstzf\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578549szh\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.458251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.458098    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578549szh\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.460542 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.460517    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578549szh\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.470309 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.470284    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstzf\" (UniqueName: \"kubernetes.io/projected/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-kube-api-access-dstzf\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578549szh\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.597492 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.597435    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:15.868253 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:15.868029    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"]
Sep 05 02:55:15.870459 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:55:15.870434    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae95eae_d5ed_4f22_8e12_9e29b293a91a.slice/crio-7418f657c5792589538c8011b0073f94292a33fd9e8fd6235c59710d693e2cf9 WatchSource:0}: Error finding container 7418f657c5792589538c8011b0073f94292a33fd9e8fd6235c59710d693e2cf9: Status 404 returned error can't find the container with id 7418f657c5792589538c8011b0073f94292a33fd9e8fd6235c59710d693e2cf9
Sep 05 02:55:16.857039 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:16.857008    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" event={"ID":"5ae95eae-d5ed-4f22-8e12-9e29b293a91a","Type":"ContainerStarted","Data":"e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062"}
Sep 05 02:55:16.857464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:16.857061    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" event={"ID":"5ae95eae-d5ed-4f22-8e12-9e29b293a91a","Type":"ContainerStarted","Data":"7418f657c5792589538c8011b0073f94292a33fd9e8fd6235c59710d693e2cf9"}
Sep 05 02:55:16.857464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:16.857098    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:47.863806 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:47.863726    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:55:47.884411 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:47.884359    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podStartSLOduration=32.88434209 podStartE2EDuration="32.88434209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:55:16.876429682 +0000 UTC m=+2227.075792675" watchObservedRunningTime="2026-09-05 02:55:47.88434209 +0000 UTC m=+2258.083705082"
Sep 05 02:55:52.630437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:52.630397    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"]
Sep 05 02:55:57.864136 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:55:57.864095    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:56:07.654977 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:07.654940    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main" containerID="cri-o://e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062" gracePeriod=45
Sep 05 02:56:07.863793 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:07.863754    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:56:17.864056 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:17.864016    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:56:17.864473 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:17.864135    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:56:27.863794 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:27.863756    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:56:37.902852 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:37.902825    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:56:38.047236 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.047206    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dstzf\" (UniqueName: \"kubernetes.io/projected/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-kube-api-access-dstzf\") pod \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") "
Sep 05 02:56:38.047402 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.047308    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-tls-certs\") pod \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\" (UID: \"5ae95eae-d5ed-4f22-8e12-9e29b293a91a\") "
Sep 05 02:56:38.049525 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.049493    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ae95eae-d5ed-4f22-8e12-9e29b293a91a" (UID: "5ae95eae-d5ed-4f22-8e12-9e29b293a91a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:56:38.049672 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.049555    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-kube-api-access-dstzf" (OuterVolumeSpecName: "kube-api-access-dstzf") pod "5ae95eae-d5ed-4f22-8e12-9e29b293a91a" (UID: "5ae95eae-d5ed-4f22-8e12-9e29b293a91a"). InnerVolumeSpecName "kube-api-access-dstzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:56:38.148375 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.148344    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:56:38.148375 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.148371    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstzf\" (UniqueName: \"kubernetes.io/projected/5ae95eae-d5ed-4f22-8e12-9e29b293a91a-kube-api-access-dstzf\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:56:38.239306 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.239271    2553 generic.go:356] "Generic (PLEG): container finished" podID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerID="e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062" exitCode=0
Sep 05 02:56:38.239471 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.239326    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" event={"ID":"5ae95eae-d5ed-4f22-8e12-9e29b293a91a","Type":"ContainerDied","Data":"e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062"}
Sep 05 02:56:38.239471 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.239367    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" event={"ID":"5ae95eae-d5ed-4f22-8e12-9e29b293a91a","Type":"ContainerDied","Data":"7418f657c5792589538c8011b0073f94292a33fd9e8fd6235c59710d693e2cf9"}
Sep 05 02:56:38.239471 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.239387    2553 scope.go:117] "RemoveContainer" containerID="e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062"
Sep 05 02:56:38.239471 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.239327    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"
Sep 05 02:56:38.254076 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.254053    2553 scope.go:117] "RemoveContainer" containerID="e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062"
Sep 05 02:56:38.254498 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:56:38.254428    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062\": container with ID starting with e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062 not found: ID does not exist" containerID="e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062"
Sep 05 02:56:38.254498 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.254464    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062"} err="failed to get container status \"e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062\": rpc error: code = NotFound desc = could not find container \"e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062\": container with ID starting with e2663eb18744f250c8559dc08f0e74ac18fbbca3c23f02ebbbf37519ea1b9062 not found: ID does not exist"
Sep 05 02:56:38.267070 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.267038    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"]
Sep 05 02:56:38.270319 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.270292    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh"]
Sep 05 02:56:38.392972 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.392901    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" path="/var/lib/kubelet/pods/5ae95eae-d5ed-4f22-8e12-9e29b293a91a/volumes"
Sep 05 02:56:38.862932 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:38.862894    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578549szh" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.50:9003\" within 1s: context deadline exceeded"
Sep 05 02:56:38.863099 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:56:38.862977    2553 logging.go:55] [core] [Channel #587 SubChannel #588]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.50:9003", ServerName: "10.132.0.50:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.50:9003: operation was canceled"
Sep 05 02:56:52.145789 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.145750    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"]
Sep 05 02:56:52.146386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.146196    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main"
Sep 05 02:56:52.146386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.146212    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main"
Sep 05 02:56:52.146386 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.146287    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae95eae-d5ed-4f22-8e12-9e29b293a91a" containerName="main"
Sep 05 02:56:52.149074 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.149053    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.152677 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.152650    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-8nwmc\""
Sep 05 02:56:52.152833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.152686    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-86nxw\""
Sep 05 02:56:52.152833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.152709    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 05 02:56:52.152833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.152771    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 05 02:56:52.152833 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.152687    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 05 02:56:52.161282 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.161257    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"]
Sep 05 02:56:52.260255 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.260227    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7c5c4-b3cc-4841-9308-9542b4439167-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.260417 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.260294    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/0cc7c5c4-b3cc-4841-9308-9542b4439167-kube-api-access-4sh74\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.361555 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.361520    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/0cc7c5c4-b3cc-4841-9308-9542b4439167-kube-api-access-4sh74\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.361744 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.361696    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7c5c4-b3cc-4841-9308-9542b4439167-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.364157 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.364129    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7c5c4-b3cc-4841-9308-9542b4439167-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.370381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.370350    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/0cc7c5c4-b3cc-4841-9308-9542b4439167-kube-api-access-4sh74\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.459792 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.459711    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:52.738637 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:52.738573    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"]
Sep 05 02:56:52.741316 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:56:52.741287    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc7c5c4_b3cc_4841_9308_9542b4439167.slice/crio-40bd7490d9a44da499a8d54f8b989e72e0b7edc52046ee40125ac6023cab522f WatchSource:0}: Error finding container 40bd7490d9a44da499a8d54f8b989e72e0b7edc52046ee40125ac6023cab522f: Status 404 returned error can't find the container with id 40bd7490d9a44da499a8d54f8b989e72e0b7edc52046ee40125ac6023cab522f
Sep 05 02:56:53.318839 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:53.318804    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" event={"ID":"0cc7c5c4-b3cc-4841-9308-9542b4439167","Type":"ContainerStarted","Data":"6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f"}
Sep 05 02:56:53.319267 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:53.318859    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" event={"ID":"0cc7c5c4-b3cc-4841-9308-9542b4439167","Type":"ContainerStarted","Data":"40bd7490d9a44da499a8d54f8b989e72e0b7edc52046ee40125ac6023cab522f"}
Sep 05 02:56:53.319267 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:53.318922    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:56:53.340203 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:56:53.340155    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podStartSLOduration=1.3401380010000001 podStartE2EDuration="1.340138001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:53.338214437 +0000 UTC m=+2323.537577429" watchObservedRunningTime="2026-09-05 02:56:53.340138001 +0000 UTC m=+2323.539500995"
Sep 05 02:57:24.325369 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:24.325337    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:57:36.803431 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.803386    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"]
Sep 05 02:57:36.807209 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.807190    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:36.810942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.810921    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-jtnc9\""
Sep 05 02:57:36.810942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.810932    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 05 02:57:36.811129 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.810923    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 05 02:57:36.811244 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.811225    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 05 02:57:36.819493 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.819468    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"]
Sep 05 02:57:36.943156 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.943126    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-home\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:36.943362 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.943168    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-tmp-dir\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:36.943362 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.943191    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-dshm\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:36.943362 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.943280    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-model-cache\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:36.943524 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.943357    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wffc6\" (UniqueName: \"kubernetes.io/projected/8ed6c252-8411-4868-83de-84b34b1b9e87-kube-api-access-wffc6\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:36.943524 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:36.943414    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed6c252-8411-4868-83de-84b34b1b9e87-tls-certs\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.043904 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.043869    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-dshm\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044043 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.043911    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-model-cache\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044043 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.043950    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wffc6\" (UniqueName: \"kubernetes.io/projected/8ed6c252-8411-4868-83de-84b34b1b9e87-kube-api-access-wffc6\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044043 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.043982    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed6c252-8411-4868-83de-84b34b1b9e87-tls-certs\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044043 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.044011    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-home\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044043 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.044035    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-tmp-dir\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044402 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.044374    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-tmp-dir\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044402 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.044394    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-model-cache\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.044502 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.044435    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-home\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.046211 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.046193    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-dshm\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.046613 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.046593    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed6c252-8411-4868-83de-84b34b1b9e87-tls-certs\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.051893 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.051871    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wffc6\" (UniqueName: \"kubernetes.io/projected/8ed6c252-8411-4868-83de-84b34b1b9e87-kube-api-access-wffc6\") pod \"late-v2-kserve-86766546f-zww24\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.118949 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.118881    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:37.396472 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.396442    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"]
Sep 05 02:57:37.397945 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:57:37.397918    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed6c252_8411_4868_83de_84b34b1b9e87.slice/crio-a835068a36b801eecfd4e72e4ba0b16ea5e2a2adca304704c997b83109ad3c56 WatchSource:0}: Error finding container a835068a36b801eecfd4e72e4ba0b16ea5e2a2adca304704c997b83109ad3c56: Status 404 returned error can't find the container with id a835068a36b801eecfd4e72e4ba0b16ea5e2a2adca304704c997b83109ad3c56
Sep 05 02:57:37.531172 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.531140    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" event={"ID":"8ed6c252-8411-4868-83de-84b34b1b9e87","Type":"ContainerStarted","Data":"fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a"}
Sep 05 02:57:37.531355 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.531190    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" event={"ID":"8ed6c252-8411-4868-83de-84b34b1b9e87","Type":"ContainerStarted","Data":"a835068a36b801eecfd4e72e4ba0b16ea5e2a2adca304704c997b83109ad3c56"}
Sep 05 02:57:37.549526 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:37.549454    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" podStartSLOduration=1.5494311189999999 podStartE2EDuration="1.549431119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:57:37.547704902 +0000 UTC m=+2367.747067902" watchObservedRunningTime="2026-09-05 02:57:37.549431119 +0000 UTC m=+2367.748794118"
Sep 05 02:57:38.119550 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:38.119515    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:47.119181 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:47.119143    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:47.132348 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:47.132320    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:47.588098 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:47.588067    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:57:54.040942 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:54.040899    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"]
Sep 05 02:57:54.325357 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:57:54.325277    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:58:04.325908 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:04.325865    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:58:09.062006 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:09.061960    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main" containerID="cri-o://6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f" gracePeriod=45
Sep 05 02:58:14.325046 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:14.324997    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:58:14.325428 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:14.325115    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:58:20.359108 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:20.359072    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"]
Sep 05 02:58:24.325232 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:24.325186    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:58:34.325784 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:34.325693    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:58:35.378776 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.378714    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" podUID="8ed6c252-8411-4868-83de-84b34b1b9e87" containerName="main" containerID="cri-o://fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a" gracePeriod=45
Sep 05 02:58:35.624137 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.624113    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:58:35.736837 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.736769    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-dshm\") pod \"8ed6c252-8411-4868-83de-84b34b1b9e87\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") "
Sep 05 02:58:35.736837 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.736804    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wffc6\" (UniqueName: \"kubernetes.io/projected/8ed6c252-8411-4868-83de-84b34b1b9e87-kube-api-access-wffc6\") pod \"8ed6c252-8411-4868-83de-84b34b1b9e87\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") "
Sep 05 02:58:35.736837 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.736844    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-model-cache\") pod \"8ed6c252-8411-4868-83de-84b34b1b9e87\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") "
Sep 05 02:58:35.737116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.736877    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-home\") pod \"8ed6c252-8411-4868-83de-84b34b1b9e87\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") "
Sep 05 02:58:35.737116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.736913    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed6c252-8411-4868-83de-84b34b1b9e87-tls-certs\") pod \"8ed6c252-8411-4868-83de-84b34b1b9e87\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") "
Sep 05 02:58:35.737116 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737007    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-tmp-dir\") pod \"8ed6c252-8411-4868-83de-84b34b1b9e87\" (UID: \"8ed6c252-8411-4868-83de-84b34b1b9e87\") "
Sep 05 02:58:35.737267 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737186    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ed6c252-8411-4868-83de-84b34b1b9e87" (UID: "8ed6c252-8411-4868-83de-84b34b1b9e87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:35.737267 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737205    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-home" (OuterVolumeSpecName: "home") pod "8ed6c252-8411-4868-83de-84b34b1b9e87" (UID: "8ed6c252-8411-4868-83de-84b34b1b9e87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:35.737355 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737261    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ed6c252-8411-4868-83de-84b34b1b9e87" (UID: "8ed6c252-8411-4868-83de-84b34b1b9e87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:35.737420 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737403    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:35.737464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737426    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:35.737464 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.737440    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:35.739120 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.739099    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-dshm" (OuterVolumeSpecName: "dshm") pod "8ed6c252-8411-4868-83de-84b34b1b9e87" (UID: "8ed6c252-8411-4868-83de-84b34b1b9e87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:35.739228 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.739132    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed6c252-8411-4868-83de-84b34b1b9e87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ed6c252-8411-4868-83de-84b34b1b9e87" (UID: "8ed6c252-8411-4868-83de-84b34b1b9e87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:58:35.739228 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.739176    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed6c252-8411-4868-83de-84b34b1b9e87-kube-api-access-wffc6" (OuterVolumeSpecName: "kube-api-access-wffc6") pod "8ed6c252-8411-4868-83de-84b34b1b9e87" (UID: "8ed6c252-8411-4868-83de-84b34b1b9e87"). InnerVolumeSpecName "kube-api-access-wffc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:58:35.807822 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.807785    2553 generic.go:356] "Generic (PLEG): container finished" podID="8ed6c252-8411-4868-83de-84b34b1b9e87" containerID="fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a" exitCode=0
Sep 05 02:58:35.807966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.807835    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"
Sep 05 02:58:35.807966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.807868    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" event={"ID":"8ed6c252-8411-4868-83de-84b34b1b9e87","Type":"ContainerDied","Data":"fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a"}
Sep 05 02:58:35.807966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.807936    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" event={"ID":"8ed6c252-8411-4868-83de-84b34b1b9e87","Type":"ContainerDied","Data":"a835068a36b801eecfd4e72e4ba0b16ea5e2a2adca304704c997b83109ad3c56"}
Sep 05 02:58:35.807966 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.807963    2553 scope.go:117] "RemoveContainer" containerID="fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a"
Sep 05 02:58:35.818817 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.818780    2553 scope.go:117] "RemoveContainer" containerID="fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a"
Sep 05 02:58:35.819184 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:58:35.819162    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a\": container with ID starting with fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a not found: ID does not exist" containerID="fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a"
Sep 05 02:58:35.819260 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.819193    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a"} err="failed to get container status \"fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a\": rpc error: code = NotFound desc = could not find container \"fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a\": container with ID starting with fa7812ddc0433c096a995e64930f4c6630bdce0f10b529098533a380c82eb50a not found: ID does not exist"
Sep 05 02:58:35.831495 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.831469    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"]
Sep 05 02:58:35.835320 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.835299    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24"]
Sep 05 02:58:35.838866 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.838839    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed6c252-8411-4868-83de-84b34b1b9e87-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:35.838866 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.838867    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wffc6\" (UniqueName: \"kubernetes.io/projected/8ed6c252-8411-4868-83de-84b34b1b9e87-kube-api-access-wffc6\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:35.839005 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:35.838878    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed6c252-8411-4868-83de-84b34b1b9e87-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:36.393890 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:36.393856    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed6c252-8411-4868-83de-84b34b1b9e87" path="/var/lib/kubelet/pods/8ed6c252-8411-4868-83de-84b34b1b9e87/volumes"
Sep 05 02:58:36.577877 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:36.577820    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zww24" podUID="8ed6c252-8411-4868-83de-84b34b1b9e87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Sep 05 02:58:39.307239 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.307218    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:58:39.465812 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.465718    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7c5c4-b3cc-4841-9308-9542b4439167-tls-certs\") pod \"0cc7c5c4-b3cc-4841-9308-9542b4439167\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") "
Sep 05 02:58:39.465812 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.465767    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/0cc7c5c4-b3cc-4841-9308-9542b4439167-kube-api-access-4sh74\") pod \"0cc7c5c4-b3cc-4841-9308-9542b4439167\" (UID: \"0cc7c5c4-b3cc-4841-9308-9542b4439167\") "
Sep 05 02:58:39.467922 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.467884    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc7c5c4-b3cc-4841-9308-9542b4439167-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cc7c5c4-b3cc-4841-9308-9542b4439167" (UID: "0cc7c5c4-b3cc-4841-9308-9542b4439167"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:58:39.468039 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.467974    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc7c5c4-b3cc-4841-9308-9542b4439167-kube-api-access-4sh74" (OuterVolumeSpecName: "kube-api-access-4sh74") pod "0cc7c5c4-b3cc-4841-9308-9542b4439167" (UID: "0cc7c5c4-b3cc-4841-9308-9542b4439167"). InnerVolumeSpecName "kube-api-access-4sh74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:58:39.566814 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.566782    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7c5c4-b3cc-4841-9308-9542b4439167-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:39.566814 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.566810    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/0cc7c5c4-b3cc-4841-9308-9542b4439167-kube-api-access-4sh74\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 02:58:39.828146 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.828110    2553 generic.go:356] "Generic (PLEG): container finished" podID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerID="6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f" exitCode=0
Sep 05 02:58:39.828334 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.828161    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"
Sep 05 02:58:39.828334 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.828179    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" event={"ID":"0cc7c5c4-b3cc-4841-9308-9542b4439167","Type":"ContainerDied","Data":"6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f"}
Sep 05 02:58:39.828334 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.828252    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98" event={"ID":"0cc7c5c4-b3cc-4841-9308-9542b4439167","Type":"ContainerDied","Data":"40bd7490d9a44da499a8d54f8b989e72e0b7edc52046ee40125ac6023cab522f"}
Sep 05 02:58:39.828334 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.828294    2553 scope.go:117] "RemoveContainer" containerID="6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f"
Sep 05 02:58:39.837124 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.837104    2553 scope.go:117] "RemoveContainer" containerID="6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f"
Sep 05 02:58:39.837381 ip-10-0-143-8 kubenswrapper[2553]: E0905 02:58:39.837361    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f\": container with ID starting with 6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f not found: ID does not exist" containerID="6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f"
Sep 05 02:58:39.837432 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.837389    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f"} err="failed to get container status \"6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f\": rpc error: code = NotFound desc = could not find container \"6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f\": container with ID starting with 6fc25691d24a3e8532c3f1da0499632a335a307b8859d47f39020e6e6b051a4f not found: ID does not exist"
Sep 05 02:58:39.850449 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.850425    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"]
Sep 05 02:58:39.856182 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:39.856159    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6hs98"]
Sep 05 02:58:40.393141 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:40.393105    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" path="/var/lib/kubelet/pods/0cc7c5c4-b3cc-4841-9308-9542b4439167/volumes"
Sep 05 02:58:58.100488 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.100451    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"]
Sep 05 02:58:58.101063 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.101045    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed6c252-8411-4868-83de-84b34b1b9e87" containerName="main"
Sep 05 02:58:58.101130 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.101067    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed6c252-8411-4868-83de-84b34b1b9e87" containerName="main"
Sep 05 02:58:58.101130 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.101084    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main"
Sep 05 02:58:58.101130 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.101094    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main"
Sep 05 02:58:58.101226 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.101204    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed6c252-8411-4868-83de-84b34b1b9e87" containerName="main"
Sep 05 02:58:58.101226 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.101215    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc7c5c4-b3cc-4841-9308-9542b4439167" containerName="main"
Sep 05 02:58:58.108862 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.108836    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.111403 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.111346    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-dshm\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.111847 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.111697    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 05 02:58:58.111965 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.111842    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-home\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.112036 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.111961    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.112036 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.111997    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.112149 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.112035    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mff\" (UniqueName: \"kubernetes.io/projected/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-kube-api-access-f2mff\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.112149 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.112116    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.112680 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.112657    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 05 02:58:58.112803 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.112661    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-6f8ws\""
Sep 05 02:58:58.112993 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.112971    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 05 02:58:58.115258 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.115236    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"]
Sep 05 02:58:58.213094 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213060    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-home\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213239 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213105    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213239 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213265    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mff\" (UniqueName: \"kubernetes.io/projected/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-kube-api-access-f2mff\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213371 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213336    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213476 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213396    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-dshm\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213528 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213500    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213604 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-home\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.213646 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.213609    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.215515 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.215494    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-dshm\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.215970 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.215947    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.220803 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.220783    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mff\" (UniqueName: \"kubernetes.io/projected/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-kube-api-access-f2mff\") pod \"delw-v1-kserve-6cf8cbb847-bjmmx\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.420737 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.420670    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:58:58.704466 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.704437    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"]
Sep 05 02:58:58.706906 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:58:58.706875    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed4ccf4_99ea_40e7_9b12_fae1c9ce7c23.slice/crio-f400134d7608fd42a93ad4646eccac72ff35f608cc25d485f1ae0af71a0f2e47 WatchSource:0}: Error finding container f400134d7608fd42a93ad4646eccac72ff35f608cc25d485f1ae0af71a0f2e47: Status 404 returned error can't find the container with id f400134d7608fd42a93ad4646eccac72ff35f608cc25d485f1ae0af71a0f2e47
Sep 05 02:58:58.922872 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.922821    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx" event={"ID":"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23","Type":"ContainerStarted","Data":"36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da"}
Sep 05 02:58:58.923056 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.922900    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx" event={"ID":"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23","Type":"ContainerStarted","Data":"f400134d7608fd42a93ad4646eccac72ff35f608cc25d485f1ae0af71a0f2e47"}
Sep 05 02:58:58.942995 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:58.942914    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx" podStartSLOduration=0.942899198 podStartE2EDuration="942.899198ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:58:58.941520785 +0000 UTC m=+2449.140883778" watchObservedRunningTime="2026-09-05 02:58:58.942899198 +0000 UTC m=+2449.142262190"
Sep 05 02:58:59.421251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:58:59.421220    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:59:08.421381 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:08.421347    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:59:08.433417 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:08.433396    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:59:08.980629 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:08.980601    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 02:59:20.903079 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.903016    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"]
Sep 05 02:59:20.906630 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.906607    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:20.909245 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.909223    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 05 02:59:20.910424 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.910394    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 05 02:59:20.910424 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.910408    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-vvcb4\""
Sep 05 02:59:20.910618 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.910490    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 05 02:59:20.915828 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.915807    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"]
Sep 05 02:59:20.995285 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.995254    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:20.995437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.995295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:20.995437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.995319    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pmm\" (UniqueName: \"kubernetes.io/projected/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-kube-api-access-96pmm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:20.995437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.995363    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:20.995437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.995406    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:20.995437 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:20.995433    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.096825 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.096797    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.096954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.096831    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pmm\" (UniqueName: \"kubernetes.io/projected/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-kube-api-access-96pmm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.096954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.096852    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.096954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.096878    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.096954 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.096902    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.097182 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.096985    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.097279 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.097257    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.097334 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.097310    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.097379 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.097360    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.099128 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.099104    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.099524 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.099501    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.105332 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.105308    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pmm\" (UniqueName: \"kubernetes.io/projected/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-kube-api-access-96pmm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.217507 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.217432    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:21.344928 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.344895    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"]
Sep 05 02:59:21.348609 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.348589    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.350978 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.350954    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-mzrsf\""
Sep 05 02:59:21.357136 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.357116    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"]
Sep 05 02:59:21.399758 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.399734    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.399855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.399781    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qscb\" (UniqueName: \"kubernetes.io/projected/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-kube-api-access-6qscb\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.488326 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.488302    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"]
Sep 05 02:59:21.490304 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:59:21.490281    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5e8b85_e83c_4eff_9f19_d3322427a1ce.slice/crio-2cac1bb94b0c220460f07587cfc41a489edb1710df00551c51f5bbbf5cbe2d7c WatchSource:0}: Error finding container 2cac1bb94b0c220460f07587cfc41a489edb1710df00551c51f5bbbf5cbe2d7c: Status 404 returned error can't find the container with id 2cac1bb94b0c220460f07587cfc41a489edb1710df00551c51f5bbbf5cbe2d7c
Sep 05 02:59:21.500939 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.500915    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qscb\" (UniqueName: \"kubernetes.io/projected/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-kube-api-access-6qscb\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.501042 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.501027    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.503606 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.503588    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.509413 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.509392    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qscb\" (UniqueName: \"kubernetes.io/projected/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-kube-api-access-6qscb\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.660687 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.660656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:21.929251 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:21.929225    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"]
Sep 05 02:59:21.931752 ip-10-0-143-8 kubenswrapper[2553]: W0905 02:59:21.931728    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0122b7c9_24dc_40e8_ad62_0d5a5ca56abd.slice/crio-cf1ceced48b1c7c99e8b07c9e5443042e217ed29ffcbd0d51a57e915e7a62312 WatchSource:0}: Error finding container cf1ceced48b1c7c99e8b07c9e5443042e217ed29ffcbd0d51a57e915e7a62312: Status 404 returned error can't find the container with id cf1ceced48b1c7c99e8b07c9e5443042e217ed29ffcbd0d51a57e915e7a62312
Sep 05 02:59:22.031032 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.030992    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" event={"ID":"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd","Type":"ContainerStarted","Data":"81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb"}
Sep 05 02:59:22.031200 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.031064    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" event={"ID":"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd","Type":"ContainerStarted","Data":"cf1ceced48b1c7c99e8b07c9e5443042e217ed29ffcbd0d51a57e915e7a62312"}
Sep 05 02:59:22.031200 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.031182    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 02:59:22.032751 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.032727    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" event={"ID":"1c5e8b85-e83c-4eff-9f19-d3322427a1ce","Type":"ContainerStarted","Data":"cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"}
Sep 05 02:59:22.032855 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.032776    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" event={"ID":"1c5e8b85-e83c-4eff-9f19-d3322427a1ce","Type":"ContainerStarted","Data":"2cac1bb94b0c220460f07587cfc41a489edb1710df00551c51f5bbbf5cbe2d7c"}
Sep 05 02:59:22.049983 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.049939    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podStartSLOduration=1.049924323 podStartE2EDuration="1.049924323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:59:22.048292719 +0000 UTC m=+2472.247655725" watchObservedRunningTime="2026-09-05 02:59:22.049924323 +0000 UTC m=+2472.249287306"
Sep 05 02:59:22.069979 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.069921    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" podStartSLOduration=2.0699008 podStartE2EDuration="2.0699008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:59:22.06858078 +0000 UTC m=+2472.267943773" watchObservedRunningTime="2026-09-05 02:59:22.0699008 +0000 UTC m=+2472.269263794"
Sep 05 02:59:22.217764 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:22.217688    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:31.218544 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:31.218507    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:31.230391 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:31.230364    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:32.091372 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:32.091341    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 02:59:53.039530 ip-10-0-143-8 kubenswrapper[2553]: I0905 02:59:53.039499    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 03:00:08.122386 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:08.122301    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"]
Sep 05 03:00:08.130627 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:08.130545    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"]
Sep 05 03:00:08.140170 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:00:08.140133    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:00:08.140170 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:00:08.140170 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:00:08.140170 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339" cmd=["/bin/sleep","15"]
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:08.140192    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" message="" err=<
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:00:08.140208    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" podUID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" containerName="main" containerID="cri-o://cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"
Sep 05 03:00:08.140467 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:08.140250    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" podUID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" containerName="main" containerID="cri-o://cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339" gracePeriod=60
Sep 05 03:00:08.883206 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:08.883182    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 03:00:09.009022 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.008947    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tls-certs\") pod \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") "
Sep 05 03:00:09.009022 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.008997    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-model-cache\") pod \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") "
Sep 05 03:00:09.009022 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009020    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tmp-dir\") pod \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") "
Sep 05 03:00:09.009270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009041    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-home\") pod \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") "
Sep 05 03:00:09.009270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009062    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-dshm\") pod \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") "
Sep 05 03:00:09.009270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009089    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96pmm\" (UniqueName: \"kubernetes.io/projected/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-kube-api-access-96pmm\") pod \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\" (UID: \"1c5e8b85-e83c-4eff-9f19-d3322427a1ce\") "
Sep 05 03:00:09.009270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009256    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c5e8b85-e83c-4eff-9f19-d3322427a1ce" (UID: "1c5e8b85-e83c-4eff-9f19-d3322427a1ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:09.009505 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009303    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-home" (OuterVolumeSpecName: "home") pod "1c5e8b85-e83c-4eff-9f19-d3322427a1ce" (UID: "1c5e8b85-e83c-4eff-9f19-d3322427a1ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:09.009505 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009362    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c5e8b85-e83c-4eff-9f19-d3322427a1ce" (UID: "1c5e8b85-e83c-4eff-9f19-d3322427a1ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:09.009505 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009423    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:09.009505 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.009440    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:09.011242 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.011219    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-dshm" (OuterVolumeSpecName: "dshm") pod "1c5e8b85-e83c-4eff-9f19-d3322427a1ce" (UID: "1c5e8b85-e83c-4eff-9f19-d3322427a1ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:09.011328 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.011246    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c5e8b85-e83c-4eff-9f19-d3322427a1ce" (UID: "1c5e8b85-e83c-4eff-9f19-d3322427a1ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:00:09.011366 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.011333    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-kube-api-access-96pmm" (OuterVolumeSpecName: "kube-api-access-96pmm") pod "1c5e8b85-e83c-4eff-9f19-d3322427a1ce" (UID: "1c5e8b85-e83c-4eff-9f19-d3322427a1ce"). InnerVolumeSpecName "kube-api-access-96pmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:00:09.110845 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.110815    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:09.110845 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.110843    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:09.110993 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.110851    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:09.110993 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.110859    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96pmm\" (UniqueName: \"kubernetes.io/projected/1c5e8b85-e83c-4eff-9f19-d3322427a1ce-kube-api-access-96pmm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:09.260445 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.260374    2553 generic.go:356] "Generic (PLEG): container finished" podID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" containerID="cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339" exitCode=0
Sep 05 03:00:09.260445 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.260434    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"
Sep 05 03:00:09.260924 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.260464    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" event={"ID":"1c5e8b85-e83c-4eff-9f19-d3322427a1ce","Type":"ContainerDied","Data":"cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"}
Sep 05 03:00:09.260924 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.260534    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56" event={"ID":"1c5e8b85-e83c-4eff-9f19-d3322427a1ce","Type":"ContainerDied","Data":"2cac1bb94b0c220460f07587cfc41a489edb1710df00551c51f5bbbf5cbe2d7c"}
Sep 05 03:00:09.260924 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.260588    2553 scope.go:117] "RemoveContainer" containerID="cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"
Sep 05 03:00:09.270242 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.270224    2553 scope.go:117] "RemoveContainer" containerID="cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"
Sep 05 03:00:09.270534 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:00:09.270487    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339\": container with ID starting with cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339 not found: ID does not exist" containerID="cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"
Sep 05 03:00:09.270534 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.270516    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339"} err="failed to get container status \"cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339\": rpc error: code = NotFound desc = could not find container \"cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339\": container with ID starting with cf155a77b3c0bba57bcda8e8651d8b60b84d7b7429871df6994fea334c080339 not found: ID does not exist"
Sep 05 03:00:09.281955 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.281931    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"]
Sep 05 03:00:09.287507 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:09.287486    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-gxf56"]
Sep 05 03:00:10.392539 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:10.392501    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" path="/var/lib/kubelet/pods/1c5e8b85-e83c-4eff-9f19-d3322427a1ce/volumes"
Sep 05 03:00:13.038876 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:13.038839    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:00:22.574841 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:22.574803    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"]
Sep 05 03:00:23.039205 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:23.039169    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:00:23.149335 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:23.149304    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main" containerID="cri-o://81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb" gracePeriod=45
Sep 05 03:00:33.038965 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:33.038921    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:00:33.039477 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:33.039018    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 03:00:37.596555 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.596505    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx" podUID="4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" containerName="main" containerID="cri-o://36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da" gracePeriod=45
Sep 05 03:00:37.605517 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:00:37.605487    2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed4ccf4_99ea_40e7_9b12_fae1c9ce7c23.slice/crio-36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da.scope\": RecentStats: unable to find data in memory cache]"
Sep 05 03:00:37.844245 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.844217    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 03:00:37.854701 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854650    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-dshm\") pod \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") "
Sep 05 03:00:37.854701 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854689    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2mff\" (UniqueName: \"kubernetes.io/projected/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-kube-api-access-f2mff\") pod \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") "
Sep 05 03:00:37.854870 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854730    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tmp-dir\") pod \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") "
Sep 05 03:00:37.854870 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854756    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tls-certs\") pod \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") "
Sep 05 03:00:37.854870 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854779    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-model-cache\") pod \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") "
Sep 05 03:00:37.854870 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854804    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-home\") pod \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\" (UID: \"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23\") "
Sep 05 03:00:37.855089 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.854994    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" (UID: "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:37.855089 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.855032    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" (UID: "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:37.855199 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.855093    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:37.855199 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.855146    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:37.855199 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.855175    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-home" (OuterVolumeSpecName: "home") pod "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" (UID: "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:37.856956 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.856930    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" (UID: "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:00:37.856956 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.856941    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-kube-api-access-f2mff" (OuterVolumeSpecName: "kube-api-access-f2mff") pod "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" (UID: "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23"). InnerVolumeSpecName "kube-api-access-f2mff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:00:37.857096 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.856931    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-dshm" (OuterVolumeSpecName: "dshm") pod "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" (UID: "4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:37.956062 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.956034    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:37.956062 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.956057    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:37.956216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.956067    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mff\" (UniqueName: \"kubernetes.io/projected/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-kube-api-access-f2mff\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:37.956216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:37.956077    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:38.405546 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.405515    2553 generic.go:356] "Generic (PLEG): container finished" podID="4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" containerID="36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da" exitCode=0
Sep 05 03:00:38.405688 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.405571    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"
Sep 05 03:00:38.405688 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.405604    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx" event={"ID":"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23","Type":"ContainerDied","Data":"36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da"}
Sep 05 03:00:38.405688 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.405659    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx" event={"ID":"4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23","Type":"ContainerDied","Data":"f400134d7608fd42a93ad4646eccac72ff35f608cc25d485f1ae0af71a0f2e47"}
Sep 05 03:00:38.405795 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.405689    2553 scope.go:117] "RemoveContainer" containerID="36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da"
Sep 05 03:00:38.414730 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.414715    2553 scope.go:117] "RemoveContainer" containerID="36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da"
Sep 05 03:00:38.415026 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:00:38.415005    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da\": container with ID starting with 36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da not found: ID does not exist" containerID="36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da"
Sep 05 03:00:38.415076 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.415035    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da"} err="failed to get container status \"36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da\": rpc error: code = NotFound desc = could not find container \"36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da\": container with ID starting with 36c7e42bbd0290a91db19f13c8d74028891c42750cacf4575798905974ef67da not found: ID does not exist"
Sep 05 03:00:38.422150 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.422127    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"]
Sep 05 03:00:38.425814 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:38.425795    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bjmmx"]
Sep 05 03:00:40.395367 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:40.395334    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" path="/var/lib/kubelet/pods/4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23/volumes"
Sep 05 03:00:43.039647 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:43.039602    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:00:53.039575 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.039525    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:00:53.391190 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.391164    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 03:00:53.481665 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.481633    2553 generic.go:356] "Generic (PLEG): container finished" podID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerID="81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb" exitCode=0
Sep 05 03:00:53.481802 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.481674    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"
Sep 05 03:00:53.481802 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.481710    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" event={"ID":"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd","Type":"ContainerDied","Data":"81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb"}
Sep 05 03:00:53.481802 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.481763    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl" event={"ID":"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd","Type":"ContainerDied","Data":"cf1ceced48b1c7c99e8b07c9e5443042e217ed29ffcbd0d51a57e915e7a62312"}
Sep 05 03:00:53.481802 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.481786    2553 scope.go:117] "RemoveContainer" containerID="81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb"
Sep 05 03:00:53.489274 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.489252    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qscb\" (UniqueName: \"kubernetes.io/projected/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-kube-api-access-6qscb\") pod \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") "
Sep 05 03:00:53.489404 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.489344    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-tls-certs\") pod \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\" (UID: \"0122b7c9-24dc-40e8-ad62-0d5a5ca56abd\") "
Sep 05 03:00:53.489820 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.489792    2553 scope.go:117] "RemoveContainer" containerID="81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb"
Sep 05 03:00:53.490122 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:00:53.490098    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb\": container with ID starting with 81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb not found: ID does not exist" containerID="81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb"
Sep 05 03:00:53.490219 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.490127    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb"} err="failed to get container status \"81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb\": rpc error: code = NotFound desc = could not find container \"81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb\": container with ID starting with 81f0d5e9cbd165cfe10d732b1c7280f08c0da60f241a332fce97fb47c5700cbb not found: ID does not exist"
Sep 05 03:00:53.491336 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.491306    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-kube-api-access-6qscb" (OuterVolumeSpecName: "kube-api-access-6qscb") pod "0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" (UID: "0122b7c9-24dc-40e8-ad62-0d5a5ca56abd"). InnerVolumeSpecName "kube-api-access-6qscb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:00:53.491438 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.491386    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" (UID: "0122b7c9-24dc-40e8-ad62-0d5a5ca56abd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:00:53.590505 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.590453    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:53.590505 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.590474    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qscb\" (UniqueName: \"kubernetes.io/projected/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd-kube-api-access-6qscb\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:00:53.805913 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.805885    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"]
Sep 05 03:00:53.811707 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:53.811683    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schellvpl"]
Sep 05 03:00:54.392993 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:00:54.392953    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" path="/var/lib/kubelet/pods/0122b7c9-24dc-40e8-ad62-0d5a5ca56abd/volumes"
Sep 05 03:01:11.799619 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.799582    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"]
Sep 05 03:01:11.800021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.799953    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" containerName="main"
Sep 05 03:01:11.800021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.799965    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" containerName="main"
Sep 05 03:01:11.800021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.799989    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main"
Sep 05 03:01:11.800021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.799994    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main"
Sep 05 03:01:11.800021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.800001    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" containerName="main"
Sep 05 03:01:11.800021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.800006    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" containerName="main"
Sep 05 03:01:11.800216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.800065    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c5e8b85-e83c-4eff-9f19-d3322427a1ce" containerName="main"
Sep 05 03:01:11.800216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.800073    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0122b7c9-24dc-40e8-ad62-0d5a5ca56abd" containerName="main"
Sep 05 03:01:11.800216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.800078    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed4ccf4-99ea-40e7-9b12-fae1c9ce7c23" containerName="main"
Sep 05 03:01:11.804787 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.804764    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:11.807250 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.807226    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 05 03:01:11.807383 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.807269    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 05 03:01:11.808224 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.808204    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-swzwf\""
Sep 05 03:01:11.808300 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.808208    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 05 03:01:11.813759 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.813735    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"]
Sep 05 03:01:11.933287 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.933241    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:11.933468 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.933345    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4l49\" (UniqueName: \"kubernetes.io/projected/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-kube-api-access-t4l49\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:11.933468 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.933404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:11.933468 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.933430    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:11.933468 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.933448    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:11.933629 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:11.933532    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.007602 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.007555    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"]
Sep 05 03:01:12.011102 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.011080    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.013503 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.013485    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-pptxv\""
Sep 05 03:01:12.020511 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.020488    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"]
Sep 05 03:01:12.034808 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.034787    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.034927 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.034820    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.034927 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.034846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.034927 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.034887    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.034927 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.034910    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.035156 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.034954    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4l49\" (UniqueName: \"kubernetes.io/projected/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-kube-api-access-t4l49\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.035245 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.035226    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.035297 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.035272    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.035297 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.035288    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.037200 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.037180    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.037399 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.037383    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.045159 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.045131    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4l49\" (UniqueName: \"kubernetes.io/projected/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-kube-api-access-t4l49\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.117061 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.117001    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:12.135487 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.135460    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.135606 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.135496    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfqc\" (UniqueName: \"kubernetes.io/projected/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-kube-api-access-dtfqc\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.236513 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.236475    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.236687 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.236524    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfqc\" (UniqueName: \"kubernetes.io/projected/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-kube-api-access-dtfqc\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.238758 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.238737    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.247972 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.247946    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfqc\" (UniqueName: \"kubernetes.io/projected/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-kube-api-access-dtfqc\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.320793 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.320766    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:12.393228 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:01:12.393186    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf9bdc0_b2ee_4c0c_bf05_f93db597ee6d.slice/crio-fac75810d2d3d9d2dda64b13327d0a5b0b02c83e8f4b24f9e59c22b7c8329407 WatchSource:0}: Error finding container fac75810d2d3d9d2dda64b13327d0a5b0b02c83e8f4b24f9e59c22b7c8329407: Status 404 returned error can't find the container with id fac75810d2d3d9d2dda64b13327d0a5b0b02c83e8f4b24f9e59c22b7c8329407
Sep 05 03:01:12.395125 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.395097    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"]
Sep 05 03:01:12.579746 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.579714    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" event={"ID":"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d","Type":"ContainerStarted","Data":"f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"}
Sep 05 03:01:12.579927 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.579775    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" event={"ID":"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d","Type":"ContainerStarted","Data":"fac75810d2d3d9d2dda64b13327d0a5b0b02c83e8f4b24f9e59c22b7c8329407"}
Sep 05 03:01:12.592827 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.592798    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"]
Sep 05 03:01:12.594132 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:01:12.594110    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee7ba1c_eaea_40cd_833c_8a86bb6647b1.slice/crio-60d96f5308a4547c59f850ea79e2b53609d358a1f7713682b566b72fac66b36c WatchSource:0}: Error finding container 60d96f5308a4547c59f850ea79e2b53609d358a1f7713682b566b72fac66b36c: Status 404 returned error can't find the container with id 60d96f5308a4547c59f850ea79e2b53609d358a1f7713682b566b72fac66b36c
Sep 05 03:01:12.600541 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:12.600488    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" podStartSLOduration=1.600470465 podStartE2EDuration="1.600470465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:01:12.59628591 +0000 UTC m=+2582.795648903" watchObservedRunningTime="2026-09-05 03:01:12.600470465 +0000 UTC m=+2582.799833459"
Sep 05 03:01:13.117751 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:13.117710    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:13.585715 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:13.585680    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" event={"ID":"cee7ba1c-eaea-40cd-833c-8a86bb6647b1","Type":"ContainerStarted","Data":"592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9"}
Sep 05 03:01:13.585884 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:13.585735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" event={"ID":"cee7ba1c-eaea-40cd-833c-8a86bb6647b1","Type":"ContainerStarted","Data":"60d96f5308a4547c59f850ea79e2b53609d358a1f7713682b566b72fac66b36c"}
Sep 05 03:01:13.585884 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:13.585757    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:01:13.616078 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:13.616032    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podStartSLOduration=2.616016911 podStartE2EDuration="2.616016911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:01:13.612932406 +0000 UTC m=+2583.812295399" watchObservedRunningTime="2026-09-05 03:01:13.616016911 +0000 UTC m=+2583.815379897"
Sep 05 03:01:22.117251 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:22.117215    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:22.129596 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:22.129552    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:22.641033 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:22.640999    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:01:44.592854 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:01:44.592777    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:03:33.433445 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:33.433406    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"]
Sep 05 03:03:33.440612 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:33.440584    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"]
Sep 05 03:03:33.449035 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:03:33.449000    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:03:33.449035 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:03:33.449035 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:03:33.449035 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d" cmd=["/bin/sleep","15"]
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:33.449058    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" message="" err=<
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:03:33.449085    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerName="main" containerID="cri-o://f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"
Sep 05 03:03:33.449281 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:33.449139    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerName="main" containerID="cri-o://f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d" gracePeriod=60
Sep 05 03:03:33.630845 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:33.630806    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 05 03:03:34.193826 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.193801    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:03:34.255314 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.255279    2553 generic.go:356] "Generic (PLEG): container finished" podID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerID="f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d" exitCode=0
Sep 05 03:03:34.255461 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.255318    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"
Sep 05 03:03:34.255461 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.255319    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" event={"ID":"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d","Type":"ContainerDied","Data":"f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"}
Sep 05 03:03:34.255593 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.255469    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z" event={"ID":"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d","Type":"ContainerDied","Data":"fac75810d2d3d9d2dda64b13327d0a5b0b02c83e8f4b24f9e59c22b7c8329407"}
Sep 05 03:03:34.255593 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.255491    2553 scope.go:117] "RemoveContainer" containerID="f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"
Sep 05 03:03:34.264091 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.264075    2553 scope.go:117] "RemoveContainer" containerID="f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"
Sep 05 03:03:34.264379 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:03:34.264356    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d\": container with ID starting with f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d not found: ID does not exist" containerID="f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"
Sep 05 03:03:34.264463 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.264384    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d"} err="failed to get container status \"f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d\": rpc error: code = NotFound desc = could not find container \"f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d\": container with ID starting with f939b5f68c18f76299baef523a741fef96f0d0152a91557a879657ee07a7bc9d not found: ID does not exist"
Sep 05 03:03:34.318627 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318576    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-dshm\") pod \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") "
Sep 05 03:03:34.318627 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318602    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4l49\" (UniqueName: \"kubernetes.io/projected/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-kube-api-access-t4l49\") pod \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") "
Sep 05 03:03:34.318627 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318625    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tmp-dir\") pod \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") "
Sep 05 03:03:34.318827 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318660    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-home\") pod \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") "
Sep 05 03:03:34.318827 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318688    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-model-cache\") pod \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") "
Sep 05 03:03:34.318827 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318756    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tls-certs\") pod \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\" (UID: \"3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d\") "
Sep 05 03:03:34.318981 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318935    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-home" (OuterVolumeSpecName: "home") pod "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" (UID: "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:34.318981 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.318951    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" (UID: "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:34.319060 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.319030    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" (UID: "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:34.320744 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.320717    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-dshm" (OuterVolumeSpecName: "dshm") pod "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" (UID: "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:34.320835 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.320760    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-kube-api-access-t4l49" (OuterVolumeSpecName: "kube-api-access-t4l49") pod "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" (UID: "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d"). InnerVolumeSpecName "kube-api-access-t4l49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:03:34.320835 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.320768    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" (UID: "3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:03:34.419262 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.419236    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:03:34.419262 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.419260    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:03:34.419396 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.419272    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:03:34.419396 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.419284    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4l49\" (UniqueName: \"kubernetes.io/projected/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-kube-api-access-t4l49\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:03:34.419396 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.419297    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:03:34.419396 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.419310    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:03:34.570657 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.570585    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"]
Sep 05 03:03:34.573963 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.573933    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j7d6z"]
Sep 05 03:03:34.593016 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:34.592983    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:03:36.392654 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:36.392619    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" path="/var/lib/kubelet/pods/3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d/volumes"
Sep 05 03:03:44.592583 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:44.592528    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:03:48.454697 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:48.454640    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main" containerID="cri-o://592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9" gracePeriod=45
Sep 05 03:03:54.592806 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:54.592759    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:03:54.593309 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:03:54.592882    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:04:04.593217 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:04.593176    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:04:14.593017 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:14.592977    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:04:18.707766 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.707733    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:04:18.779819 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.779797    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-tls-certs\") pod \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") "
Sep 05 03:04:18.779958 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.779831    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtfqc\" (UniqueName: \"kubernetes.io/projected/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-kube-api-access-dtfqc\") pod \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\" (UID: \"cee7ba1c-eaea-40cd-833c-8a86bb6647b1\") "
Sep 05 03:04:18.781855 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.781830    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-kube-api-access-dtfqc" (OuterVolumeSpecName: "kube-api-access-dtfqc") pod "cee7ba1c-eaea-40cd-833c-8a86bb6647b1" (UID: "cee7ba1c-eaea-40cd-833c-8a86bb6647b1"). InnerVolumeSpecName "kube-api-access-dtfqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:04:18.781949 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.781874    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cee7ba1c-eaea-40cd-833c-8a86bb6647b1" (UID: "cee7ba1c-eaea-40cd-833c-8a86bb6647b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:04:18.881424 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.881393    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:04:18.881424 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:18.881418    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfqc\" (UniqueName: \"kubernetes.io/projected/cee7ba1c-eaea-40cd-833c-8a86bb6647b1-kube-api-access-dtfqc\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:04:19.468277 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.468243    2553 generic.go:356] "Generic (PLEG): container finished" podID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerID="592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9" exitCode=0
Sep 05 03:04:19.468277 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.468277    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"
Sep 05 03:04:19.468511 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.468328    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" event={"ID":"cee7ba1c-eaea-40cd-833c-8a86bb6647b1","Type":"ContainerDied","Data":"592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9"}
Sep 05 03:04:19.468511 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.468370    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s" event={"ID":"cee7ba1c-eaea-40cd-833c-8a86bb6647b1","Type":"ContainerDied","Data":"60d96f5308a4547c59f850ea79e2b53609d358a1f7713682b566b72fac66b36c"}
Sep 05 03:04:19.468511 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.468391    2553 scope.go:117] "RemoveContainer" containerID="592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9"
Sep 05 03:04:19.477180 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.477164    2553 scope.go:117] "RemoveContainer" containerID="592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9"
Sep 05 03:04:19.477434 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:04:19.477417    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9\": container with ID starting with 592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9 not found: ID does not exist" containerID="592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9"
Sep 05 03:04:19.477480 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.477441    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9"} err="failed to get container status \"592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9\": rpc error: code = NotFound desc = could not find container \"592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9\": container with ID starting with 592c712e9aa887b009c28dda6ab541cdfb96dfa9b3ec0ff0016f543f893e52e9 not found: ID does not exist"
Sep 05 03:04:19.488945 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.488924    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"]
Sep 05 03:04:19.490679 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:19.490658    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche5l92s"]
Sep 05 03:04:20.393591 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:20.393539    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" path="/var/lib/kubelet/pods/cee7ba1c-eaea-40cd-833c-8a86bb6647b1/volumes"
Sep 05 03:04:21.482736 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.482703    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"]
Sep 05 03:04:21.483128 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.483085    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerName="main"
Sep 05 03:04:21.483128 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.483101    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerName="main"
Sep 05 03:04:21.483128 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.483109    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main"
Sep 05 03:04:21.483128 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.483115    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main"
Sep 05 03:04:21.483309 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.483173    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf9bdc0-b2ee-4c0c-bf05-f93db597ee6d" containerName="main"
Sep 05 03:04:21.483309 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.483183    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee7ba1c-eaea-40cd-833c-8a86bb6647b1" containerName="main"
Sep 05 03:04:21.487358 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.487331    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.490903 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.490869    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-b22n8\""
Sep 05 03:04:21.491777 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.491758    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 05 03:04:21.491845 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.491759    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 05 03:04:21.491895 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.491818    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 05 03:04:21.498011 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.497967    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"]
Sep 05 03:04:21.601221 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.601191    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.601397 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.601235    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.601397 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.601322    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.601397 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.601372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.601522 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.601401    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.601522 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.601437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7pfl\" (UniqueName: \"kubernetes.io/projected/9d1024ad-5913-479c-bd59-34d7e2e8aea1-kube-api-access-l7pfl\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702264 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702232    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702433 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702281    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702433 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702311    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702433 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702338    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702433 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7pfl\" (UniqueName: \"kubernetes.io/projected/9d1024ad-5913-479c-bd59-34d7e2e8aea1-kube-api-access-l7pfl\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702691 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702459    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702750 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702691    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702750 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702726    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.702851 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.702778    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.704645 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.704625    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.704933 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.704913    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.711812 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.711793    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7pfl\" (UniqueName: \"kubernetes.io/projected/9d1024ad-5913-479c-bd59-34d7e2e8aea1-kube-api-access-l7pfl\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:21.801203 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:21.801173    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:22.072763 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:22.072733    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"]
Sep 05 03:04:22.075146 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:04:22.075105    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1024ad_5913_479c_bd59_34d7e2e8aea1.slice/crio-7e154888b7dc317e42d59c4a5978f2d88c2ddb11c0123c7855e35caa3e19c8bc WatchSource:0}: Error finding container 7e154888b7dc317e42d59c4a5978f2d88c2ddb11c0123c7855e35caa3e19c8bc: Status 404 returned error can't find the container with id 7e154888b7dc317e42d59c4a5978f2d88c2ddb11c0123c7855e35caa3e19c8bc
Sep 05 03:04:22.484664 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:22.484539    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" event={"ID":"9d1024ad-5913-479c-bd59-34d7e2e8aea1","Type":"ContainerStarted","Data":"fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"}
Sep 05 03:04:22.484664 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:22.484626    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" event={"ID":"9d1024ad-5913-479c-bd59-34d7e2e8aea1","Type":"ContainerStarted","Data":"7e154888b7dc317e42d59c4a5978f2d88c2ddb11c0123c7855e35caa3e19c8bc"}
Sep 05 03:04:22.502272 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:22.502224    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" podStartSLOduration=1.502210441 podStartE2EDuration="1.502210441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:04:22.500753524 +0000 UTC m=+2772.700116524" watchObservedRunningTime="2026-09-05 03:04:22.502210441 +0000 UTC m=+2772.701573434"
Sep 05 03:04:22.801414 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:22.801388    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:31.389146 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.389115    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"]
Sep 05 03:04:31.392476 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.392457    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.396604 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.396579    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 05 03:04:31.396694 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.396626    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-bz4ww\""
Sep 05 03:04:31.396823 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.396805    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-z5f29\""
Sep 05 03:04:31.397004 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.396989    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 05 03:04:31.397089 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.396993    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 05 03:04:31.405735 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.405712    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"]
Sep 05 03:04:31.485938 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.485879    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtz5\" (UniqueName: \"kubernetes.io/projected/82cce8c1-9301-47f2-8cea-688e7ce53378-kube-api-access-hdtz5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.486102 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.486080    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cce8c1-9301-47f2-8cea-688e7ce53378-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.587293 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.587266    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cce8c1-9301-47f2-8cea-688e7ce53378-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.587408 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.587333    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtz5\" (UniqueName: \"kubernetes.io/projected/82cce8c1-9301-47f2-8cea-688e7ce53378-kube-api-access-hdtz5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.589715 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.589693    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cce8c1-9301-47f2-8cea-688e7ce53378-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.600972 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.600951    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtz5\" (UniqueName: \"kubernetes.io/projected/82cce8c1-9301-47f2-8cea-688e7ce53378-kube-api-access-hdtz5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.703306 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.703284    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:04:31.801997 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.801958    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:31.813933 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.813905    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:31.973638 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:31.973613    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"]
Sep 05 03:04:32.533398 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:32.533358    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" event={"ID":"82cce8c1-9301-47f2-8cea-688e7ce53378","Type":"ContainerStarted","Data":"f80966e66a57dc7d6f8a8ef1d90bd30fced07f4a95e10a788f7ca6fa9f1bcffc"}
Sep 05 03:04:32.533913 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:32.533417    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" event={"ID":"82cce8c1-9301-47f2-8cea-688e7ce53378","Type":"ContainerStarted","Data":"c20336b975a9858b1440dbff71cb4a987acb35f87fb0c2459e203f43cd6abebf"}
Sep 05 03:04:32.546082 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:32.546050    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:04:32.552871 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:32.552816    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" podStartSLOduration=1.55279843 podStartE2EDuration="1.55279843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:04:32.552072161 +0000 UTC m=+2782.751435156" watchObservedRunningTime="2026-09-05 03:04:32.55279843 +0000 UTC m=+2782.752161427"
Sep 05 03:04:33.538886 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:04:33.538846    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:05:04.544206 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:04.544177    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:05:37.394993 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.394959    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"]
Sep 05 03:05:37.404000 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:05:37.403968    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:05:37.404000 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:05:37.404000 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:05:37.404000 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d" cmd=["/bin/sleep","15"]
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.404015    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" message="" err=<
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:05:37.404027    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerName="main" containerID="cri-o://fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"
Sep 05 03:05:37.404304 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.404061    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerName="main" containerID="cri-o://fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d" gracePeriod=60
Sep 05 03:05:37.534632 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.534594    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 05 03:05:37.751267 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.751240    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:05:37.827371 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827337    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-model-cache\") pod \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") "
Sep 05 03:05:37.827512 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827380    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tls-certs\") pod \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") "
Sep 05 03:05:37.827512 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827432    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-dshm\") pod \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") "
Sep 05 03:05:37.827512 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827462    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tmp-dir\") pod \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") "
Sep 05 03:05:37.827662 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827615    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7pfl\" (UniqueName: \"kubernetes.io/projected/9d1024ad-5913-479c-bd59-34d7e2e8aea1-kube-api-access-l7pfl\") pod \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") "
Sep 05 03:05:37.827662 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827644    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-home\") pod \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\" (UID: \"9d1024ad-5913-479c-bd59-34d7e2e8aea1\") "
Sep 05 03:05:37.827769 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827659    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d1024ad-5913-479c-bd59-34d7e2e8aea1" (UID: "9d1024ad-5913-479c-bd59-34d7e2e8aea1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:37.827829 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827767    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d1024ad-5913-479c-bd59-34d7e2e8aea1" (UID: "9d1024ad-5913-479c-bd59-34d7e2e8aea1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:37.827979 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.827946    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-home" (OuterVolumeSpecName: "home") pod "9d1024ad-5913-479c-bd59-34d7e2e8aea1" (UID: "9d1024ad-5913-479c-bd59-34d7e2e8aea1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:37.828114 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.828024    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:05:37.828114 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.828068    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:05:37.829678 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.829650    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-dshm" (OuterVolumeSpecName: "dshm") pod "9d1024ad-5913-479c-bd59-34d7e2e8aea1" (UID: "9d1024ad-5913-479c-bd59-34d7e2e8aea1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:37.829678 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.829661    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1024ad-5913-479c-bd59-34d7e2e8aea1-kube-api-access-l7pfl" (OuterVolumeSpecName: "kube-api-access-l7pfl") pod "9d1024ad-5913-479c-bd59-34d7e2e8aea1" (UID: "9d1024ad-5913-479c-bd59-34d7e2e8aea1"). InnerVolumeSpecName "kube-api-access-l7pfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:05:37.829804 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.829683    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d1024ad-5913-479c-bd59-34d7e2e8aea1" (UID: "9d1024ad-5913-479c-bd59-34d7e2e8aea1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:05:37.853288 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.853262    2553 generic.go:356] "Generic (PLEG): container finished" podID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerID="fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d" exitCode=0
Sep 05 03:05:37.853418 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.853307    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"
Sep 05 03:05:37.853418 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.853337    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" event={"ID":"9d1024ad-5913-479c-bd59-34d7e2e8aea1","Type":"ContainerDied","Data":"fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"}
Sep 05 03:05:37.853418 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.853392    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9" event={"ID":"9d1024ad-5913-479c-bd59-34d7e2e8aea1","Type":"ContainerDied","Data":"7e154888b7dc317e42d59c4a5978f2d88c2ddb11c0123c7855e35caa3e19c8bc"}
Sep 05 03:05:37.853585 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.853430    2553 scope.go:117] "RemoveContainer" containerID="fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"
Sep 05 03:05:37.862570 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.862540    2553 scope.go:117] "RemoveContainer" containerID="fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"
Sep 05 03:05:37.862821 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:05:37.862803    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d\": container with ID starting with fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d not found: ID does not exist" containerID="fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"
Sep 05 03:05:37.862891 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.862838    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d"} err="failed to get container status \"fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d\": rpc error: code = NotFound desc = could not find container \"fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d\": container with ID starting with fea2c2eda0686e77433ee3349967a74f29a3d63222418a790e5e417fbbb9780d not found: ID does not exist"
Sep 05 03:05:37.879360 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.879339    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"]
Sep 05 03:05:37.885904 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.885879    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997gclh9"]
Sep 05 03:05:37.929270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.929223    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:05:37.929270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.929242    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7pfl\" (UniqueName: \"kubernetes.io/projected/9d1024ad-5913-479c-bd59-34d7e2e8aea1-kube-api-access-l7pfl\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:05:37.929270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.929251    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1024ad-5913-479c-bd59-34d7e2e8aea1-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:05:37.929270 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:37.929261    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1024ad-5913-479c-bd59-34d7e2e8aea1-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:05:38.394458 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:05:38.394426    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" path="/var/lib/kubelet/pods/9d1024ad-5913-479c-bd59-34d7e2e8aea1/volumes"
Sep 05 03:06:01.499682 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.499642    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"]
Sep 05 03:06:01.500205 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.500184    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerName="main"
Sep 05 03:06:01.500287 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.500209    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerName="main"
Sep 05 03:06:01.500349 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.500297    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1024ad-5913-479c-bd59-34d7e2e8aea1" containerName="main"
Sep 05 03:06:01.503209 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.503186    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.505962 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.505939    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 05 03:06:01.508361 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.508339    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 05 03:06:01.508716 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.508697    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 05 03:06:01.508825 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.508757    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-hvhmc\""
Sep 05 03:06:01.509443 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.509417    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-n8cqr\""
Sep 05 03:06:01.516314 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.516293    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"]
Sep 05 03:06:01.523435 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.523415    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26aaddea-35e7-4501-ac18-22e4ea774df7-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.523522 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.523483    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49tq\" (UniqueName: \"kubernetes.io/projected/26aaddea-35e7-4501-ac18-22e4ea774df7-kube-api-access-r49tq\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.624397 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.624359    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49tq\" (UniqueName: \"kubernetes.io/projected/26aaddea-35e7-4501-ac18-22e4ea774df7-kube-api-access-r49tq\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.624603 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.624417    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26aaddea-35e7-4501-ac18-22e4ea774df7-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.626821 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.626796    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26aaddea-35e7-4501-ac18-22e4ea774df7-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.633971 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.633940    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49tq\" (UniqueName: \"kubernetes.io/projected/26aaddea-35e7-4501-ac18-22e4ea774df7-kube-api-access-r49tq\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:01.813746 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:01.813719    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:02.091405 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:02.091381    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"]
Sep 05 03:06:02.093799 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:06:02.093769    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26aaddea_35e7_4501_ac18_22e4ea774df7.slice/crio-82d12e11c132508d36eb723ca405b87fdae917d0f4572643be863ed1f8529ecf WatchSource:0}: Error finding container 82d12e11c132508d36eb723ca405b87fdae917d0f4572643be863ed1f8529ecf: Status 404 returned error can't find the container with id 82d12e11c132508d36eb723ca405b87fdae917d0f4572643be863ed1f8529ecf
Sep 05 03:06:02.971470 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:02.971381    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" event={"ID":"26aaddea-35e7-4501-ac18-22e4ea774df7","Type":"ContainerStarted","Data":"602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1"}
Sep 05 03:06:02.971470 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:02.971465    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" event={"ID":"26aaddea-35e7-4501-ac18-22e4ea774df7","Type":"ContainerStarted","Data":"82d12e11c132508d36eb723ca405b87fdae917d0f4572643be863ed1f8529ecf"}
Sep 05 03:06:02.971899 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:02.971523    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:06:02.990414 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:02.990361    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podStartSLOduration=1.990347895 podStartE2EDuration="1.990347895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:06:02.988282352 +0000 UTC m=+2873.187645345" watchObservedRunningTime="2026-09-05 03:06:02.990347895 +0000 UTC m=+2873.189710888"
Sep 05 03:06:33.977701 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:06:33.977672    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:07:06.272866 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:06.272827    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"]
Sep 05 03:07:14.544789 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:14.544746    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:07:21.293476 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:21.293403    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main" containerID="cri-o://f80966e66a57dc7d6f8a8ef1d90bd30fced07f4a95e10a788f7ca6fa9f1bcffc" gracePeriod=45
Sep 05 03:07:24.545275 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:24.545235    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:07:34.544094 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:34.543997    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:07:34.544614 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:34.544133    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:07:44.545074 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:44.545032    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:07:51.495862 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.495824    2553 generic.go:356] "Generic (PLEG): container finished" podID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerID="f80966e66a57dc7d6f8a8ef1d90bd30fced07f4a95e10a788f7ca6fa9f1bcffc" exitCode=0
Sep 05 03:07:51.496238 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.495901    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" event={"ID":"82cce8c1-9301-47f2-8cea-688e7ce53378","Type":"ContainerDied","Data":"f80966e66a57dc7d6f8a8ef1d90bd30fced07f4a95e10a788f7ca6fa9f1bcffc"}
Sep 05 03:07:51.543988 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.543960    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:07:51.549408 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.549388    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cce8c1-9301-47f2-8cea-688e7ce53378-tls-certs\") pod \"82cce8c1-9301-47f2-8cea-688e7ce53378\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") "
Sep 05 03:07:51.549468 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.549455    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdtz5\" (UniqueName: \"kubernetes.io/projected/82cce8c1-9301-47f2-8cea-688e7ce53378-kube-api-access-hdtz5\") pod \"82cce8c1-9301-47f2-8cea-688e7ce53378\" (UID: \"82cce8c1-9301-47f2-8cea-688e7ce53378\") "
Sep 05 03:07:51.551664 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.551630    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cce8c1-9301-47f2-8cea-688e7ce53378-kube-api-access-hdtz5" (OuterVolumeSpecName: "kube-api-access-hdtz5") pod "82cce8c1-9301-47f2-8cea-688e7ce53378" (UID: "82cce8c1-9301-47f2-8cea-688e7ce53378"). InnerVolumeSpecName "kube-api-access-hdtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:07:51.551776 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.551639    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cce8c1-9301-47f2-8cea-688e7ce53378-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82cce8c1-9301-47f2-8cea-688e7ce53378" (UID: "82cce8c1-9301-47f2-8cea-688e7ce53378"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:07:51.649981 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.649939    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cce8c1-9301-47f2-8cea-688e7ce53378-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:07:51.649981 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:51.649977    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtz5\" (UniqueName: \"kubernetes.io/projected/82cce8c1-9301-47f2-8cea-688e7ce53378-kube-api-access-hdtz5\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:07:52.502322 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:52.502289    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"
Sep 05 03:07:52.502847 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:52.502298    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7" event={"ID":"82cce8c1-9301-47f2-8cea-688e7ce53378","Type":"ContainerDied","Data":"c20336b975a9858b1440dbff71cb4a987acb35f87fb0c2459e203f43cd6abebf"}
Sep 05 03:07:52.502847 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:52.502430    2553 scope.go:117] "RemoveContainer" containerID="f80966e66a57dc7d6f8a8ef1d90bd30fced07f4a95e10a788f7ca6fa9f1bcffc"
Sep 05 03:07:52.521169 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:52.521143    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"]
Sep 05 03:07:52.525160 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:52.525138    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tnn7"]
Sep 05 03:07:54.393904 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:07:54.393867    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" path="/var/lib/kubelet/pods/82cce8c1-9301-47f2-8cea-688e7ce53378/volumes"
Sep 05 03:08:15.365471 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.365438    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"]
Sep 05 03:08:15.366019 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.365885    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main"
Sep 05 03:08:15.366019 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.365899    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main"
Sep 05 03:08:15.366019 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.365969    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="82cce8c1-9301-47f2-8cea-688e7ce53378" containerName="main"
Sep 05 03:08:15.370453 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.370431    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.373108 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.373076    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 05 03:08:15.373233 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.373171    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 05 03:08:15.374120 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.373907    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 05 03:08:15.374120 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.374027    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-bzklq\""
Sep 05 03:08:15.374120 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.374059    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-5r8vj\""
Sep 05 03:08:15.382350 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.382323    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"]
Sep 05 03:08:15.452182 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.452142    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7469r\" (UniqueName: \"kubernetes.io/projected/4ca23a6b-855c-4a28-9744-37876b42cff3-kube-api-access-7469r\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.452368 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.452205    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca23a6b-855c-4a28-9744-37876b42cff3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.552791 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.552757    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca23a6b-855c-4a28-9744-37876b42cff3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.552979 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.552855    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7469r\" (UniqueName: \"kubernetes.io/projected/4ca23a6b-855c-4a28-9744-37876b42cff3-kube-api-access-7469r\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.555356 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.555327    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca23a6b-855c-4a28-9744-37876b42cff3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.561906 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.561875    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7469r\" (UniqueName: \"kubernetes.io/projected/4ca23a6b-855c-4a28-9744-37876b42cff3-kube-api-access-7469r\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.683973 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.683895    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:15.959009 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:15.958816    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"]
Sep 05 03:08:15.965685 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:08:15.965649    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca23a6b_855c_4a28_9744_37876b42cff3.slice/crio-a3bcf5dd9b5752f23311e7ead36fd8d1dbdd398c5ae63132fba9f75fb6fb56f7 WatchSource:0}: Error finding container a3bcf5dd9b5752f23311e7ead36fd8d1dbdd398c5ae63132fba9f75fb6fb56f7: Status 404 returned error can't find the container with id a3bcf5dd9b5752f23311e7ead36fd8d1dbdd398c5ae63132fba9f75fb6fb56f7
Sep 05 03:08:16.613602 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:16.613541    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" event={"ID":"4ca23a6b-855c-4a28-9744-37876b42cff3","Type":"ContainerStarted","Data":"1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50"}
Sep 05 03:08:16.613994 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:16.613621    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" event={"ID":"4ca23a6b-855c-4a28-9744-37876b42cff3","Type":"ContainerStarted","Data":"a3bcf5dd9b5752f23311e7ead36fd8d1dbdd398c5ae63132fba9f75fb6fb56f7"}
Sep 05 03:08:16.613994 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:16.613664    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:08:16.632670 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:16.632616    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" podStartSLOduration=1.632596166 podStartE2EDuration="1.632596166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:08:16.630731167 +0000 UTC m=+3006.830094161" watchObservedRunningTime="2026-09-05 03:08:16.632596166 +0000 UTC m=+3006.831959160"
Sep 05 03:08:47.619992 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:08:47.619945    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:09:10.223099 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:10.223049    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"]
Sep 05 03:09:13.978303 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:13.978258    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:09:23.978419 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:23.978373    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:09:25.243885 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:25.243835    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main" containerID="cri-o://602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1" gracePeriod=45
Sep 05 03:09:33.978216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:33.978172    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:09:33.978632 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:33.978276    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:09:43.977819 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:43.977774    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:09:53.978640 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:53.978597    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:09:55.505177 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.505114    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:09:55.644852 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.644810    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26aaddea-35e7-4501-ac18-22e4ea774df7-tls-certs\") pod \"26aaddea-35e7-4501-ac18-22e4ea774df7\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") "
Sep 05 03:09:55.645037 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.644887    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r49tq\" (UniqueName: \"kubernetes.io/projected/26aaddea-35e7-4501-ac18-22e4ea774df7-kube-api-access-r49tq\") pod \"26aaddea-35e7-4501-ac18-22e4ea774df7\" (UID: \"26aaddea-35e7-4501-ac18-22e4ea774df7\") "
Sep 05 03:09:55.647253 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.647224    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aaddea-35e7-4501-ac18-22e4ea774df7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26aaddea-35e7-4501-ac18-22e4ea774df7" (UID: "26aaddea-35e7-4501-ac18-22e4ea774df7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:09:55.647394 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.647289    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26aaddea-35e7-4501-ac18-22e4ea774df7-kube-api-access-r49tq" (OuterVolumeSpecName: "kube-api-access-r49tq") pod "26aaddea-35e7-4501-ac18-22e4ea774df7" (UID: "26aaddea-35e7-4501-ac18-22e4ea774df7"). InnerVolumeSpecName "kube-api-access-r49tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:09:55.746544 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.746512    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26aaddea-35e7-4501-ac18-22e4ea774df7-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:09:55.746544 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:55.746539    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49tq\" (UniqueName: \"kubernetes.io/projected/26aaddea-35e7-4501-ac18-22e4ea774df7-kube-api-access-r49tq\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:09:56.100888 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.100856    2553 generic.go:356] "Generic (PLEG): container finished" podID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerID="602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1" exitCode=0
Sep 05 03:09:56.101064 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.100899    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" event={"ID":"26aaddea-35e7-4501-ac18-22e4ea774df7","Type":"ContainerDied","Data":"602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1"}
Sep 05 03:09:56.101064 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.100901    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"
Sep 05 03:09:56.101064 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.100967    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh" event={"ID":"26aaddea-35e7-4501-ac18-22e4ea774df7","Type":"ContainerDied","Data":"82d12e11c132508d36eb723ca405b87fdae917d0f4572643be863ed1f8529ecf"}
Sep 05 03:09:56.101064 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.100988    2553 scope.go:117] "RemoveContainer" containerID="602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1"
Sep 05 03:09:56.109963 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.109946    2553 scope.go:117] "RemoveContainer" containerID="602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1"
Sep 05 03:09:56.110248 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:09:56.110229    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1\": container with ID starting with 602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1 not found: ID does not exist" containerID="602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1"
Sep 05 03:09:56.110287 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.110256    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1"} err="failed to get container status \"602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1\": rpc error: code = NotFound desc = could not find container \"602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1\": container with ID starting with 602a36012d54d5ef7a6abb8527a0a6b6f42c043ae4c1e6d2b49a1d6b5046bee1 not found: ID does not exist"
Sep 05 03:09:56.124376 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.124354    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"]
Sep 05 03:09:56.125839 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.125818    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schef2xwh"]
Sep 05 03:09:56.393326 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:09:56.393237    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" path="/var/lib/kubelet/pods/26aaddea-35e7-4501-ac18-22e4ea774df7/volumes"
Sep 05 03:10:18.731870 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.731833    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"]
Sep 05 03:10:18.732300 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.732275    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main"
Sep 05 03:10:18.732300 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.732295    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main"
Sep 05 03:10:18.732415 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.732402    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="26aaddea-35e7-4501-ac18-22e4ea774df7" containerName="main"
Sep 05 03:10:18.736553 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.736523    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:18.740040 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.740012    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 05 03:10:18.740214 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.740185    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-p2nn7\""
Sep 05 03:10:18.740280 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.740232    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 05 03:10:18.740334 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.740195    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-wsnts\""
Sep 05 03:10:18.740334 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.740206    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 05 03:10:18.745318 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.745294    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"]
Sep 05 03:10:18.750218 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.750180    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp2l\" (UniqueName: \"kubernetes.io/projected/1c50ede2-4f4d-4043-8f49-97abc25496c6-kube-api-access-lhp2l\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:18.750372 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.750268    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c50ede2-4f4d-4043-8f49-97abc25496c6-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:18.850967 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.850917    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp2l\" (UniqueName: \"kubernetes.io/projected/1c50ede2-4f4d-4043-8f49-97abc25496c6-kube-api-access-lhp2l\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:18.851167 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.851005    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c50ede2-4f4d-4043-8f49-97abc25496c6-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:18.853697 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.853667    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c50ede2-4f4d-4043-8f49-97abc25496c6-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:18.860074 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:18.860042    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp2l\" (UniqueName: \"kubernetes.io/projected/1c50ede2-4f4d-4043-8f49-97abc25496c6-kube-api-access-lhp2l\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:19.048260 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:19.048222    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:19.335379 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:19.335314    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"]
Sep 05 03:10:19.337808 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:10:19.337772    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c50ede2_4f4d_4043_8f49_97abc25496c6.slice/crio-f9d2efde1094d3589b63a9a925cca2ed5038f2745f11c52c7e28ecf0a367fb26 WatchSource:0}: Error finding container f9d2efde1094d3589b63a9a925cca2ed5038f2745f11c52c7e28ecf0a367fb26: Status 404 returned error can't find the container with id f9d2efde1094d3589b63a9a925cca2ed5038f2745f11c52c7e28ecf0a367fb26
Sep 05 03:10:20.219349 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:20.219312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" event={"ID":"1c50ede2-4f4d-4043-8f49-97abc25496c6","Type":"ContainerStarted","Data":"94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866"}
Sep 05 03:10:20.219833 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:20.219374    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" event={"ID":"1c50ede2-4f4d-4043-8f49-97abc25496c6","Type":"ContainerStarted","Data":"f9d2efde1094d3589b63a9a925cca2ed5038f2745f11c52c7e28ecf0a367fb26"}
Sep 05 03:10:20.219833 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:20.219404    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:20.240865 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:20.240811    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" podStartSLOduration=2.240792198 podStartE2EDuration="2.240792198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:10:20.238684791 +0000 UTC m=+3130.438047784" watchObservedRunningTime="2026-09-05 03:10:20.240792198 +0000 UTC m=+3130.440155190"
Sep 05 03:10:21.723028 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:21.722988    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"]
Sep 05 03:10:27.619930 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:27.619892    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:10:36.743688 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:36.743600    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main" containerID="cri-o://1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50" gracePeriod=45
Sep 05 03:10:37.620648 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:37.620602    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:10:47.619837 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:47.619789    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:10:47.620339 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:47.619933    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:10:51.225917 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:51.225888    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:10:57.619900 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:10:57.619856    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:11:06.988880 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:06.988855    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:11:07.061369 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.061329    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7469r\" (UniqueName: \"kubernetes.io/projected/4ca23a6b-855c-4a28-9744-37876b42cff3-kube-api-access-7469r\") pod \"4ca23a6b-855c-4a28-9744-37876b42cff3\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") "
Sep 05 03:11:07.061595 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.061382    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca23a6b-855c-4a28-9744-37876b42cff3-tls-certs\") pod \"4ca23a6b-855c-4a28-9744-37876b42cff3\" (UID: \"4ca23a6b-855c-4a28-9744-37876b42cff3\") "
Sep 05 03:11:07.063600 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.063543    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca23a6b-855c-4a28-9744-37876b42cff3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ca23a6b-855c-4a28-9744-37876b42cff3" (UID: "4ca23a6b-855c-4a28-9744-37876b42cff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:11:07.063600 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.063590    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca23a6b-855c-4a28-9744-37876b42cff3-kube-api-access-7469r" (OuterVolumeSpecName: "kube-api-access-7469r") pod "4ca23a6b-855c-4a28-9744-37876b42cff3" (UID: "4ca23a6b-855c-4a28-9744-37876b42cff3"). InnerVolumeSpecName "kube-api-access-7469r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:11:07.162548 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.162505    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7469r\" (UniqueName: \"kubernetes.io/projected/4ca23a6b-855c-4a28-9744-37876b42cff3-kube-api-access-7469r\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:11:07.162548 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.162542    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca23a6b-855c-4a28-9744-37876b42cff3-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:11:07.446628 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.446519    2553 generic.go:356] "Generic (PLEG): container finished" podID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerID="1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50" exitCode=0
Sep 05 03:11:07.446628 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.446576    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"
Sep 05 03:11:07.446628 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.446608    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" event={"ID":"4ca23a6b-855c-4a28-9744-37876b42cff3","Type":"ContainerDied","Data":"1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50"}
Sep 05 03:11:07.446921 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.446670    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv" event={"ID":"4ca23a6b-855c-4a28-9744-37876b42cff3","Type":"ContainerDied","Data":"a3bcf5dd9b5752f23311e7ead36fd8d1dbdd398c5ae63132fba9f75fb6fb56f7"}
Sep 05 03:11:07.446921 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.446698    2553 scope.go:117] "RemoveContainer" containerID="1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50"
Sep 05 03:11:07.455806 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.455788    2553 scope.go:117] "RemoveContainer" containerID="1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50"
Sep 05 03:11:07.456094 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:11:07.456074    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50\": container with ID starting with 1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50 not found: ID does not exist" containerID="1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50"
Sep 05 03:11:07.456147 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.456102    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50"} err="failed to get container status \"1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50\": rpc error: code = NotFound desc = could not find container \"1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50\": container with ID starting with 1d0f467bf3b531f6ce099d2b0489c55dcb21cbb22d9af41fd2f4ba283f070b50 not found: ID does not exist"
Sep 05 03:11:07.470842 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.470812    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"]
Sep 05 03:11:07.476374 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:07.476350    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schef5fbv"]
Sep 05 03:11:08.393359 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:08.393327    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" path="/var/lib/kubelet/pods/4ca23a6b-855c-4a28-9744-37876b42cff3/volumes"
Sep 05 03:11:28.032722 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.032676    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"]
Sep 05 03:11:28.033351 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.033070    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main"
Sep 05 03:11:28.033351 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.033081    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main"
Sep 05 03:11:28.033351 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.033141    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca23a6b-855c-4a28-9744-37876b42cff3" containerName="main"
Sep 05 03:11:28.036128 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.036102    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.040275 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.040246    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-pbf66\""
Sep 05 03:11:28.040424 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.040275    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-xk8v2\""
Sep 05 03:11:28.040424 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.040246    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 05 03:11:28.040424 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.040260    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 05 03:11:28.040424 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.040333    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 05 03:11:28.046344 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.046320    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"]
Sep 05 03:11:28.145265 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.145224    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94zg\" (UniqueName: \"kubernetes.io/projected/97979f66-3c4c-4410-b251-bb924552009c-kube-api-access-q94zg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.145459 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.145301    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97979f66-3c4c-4410-b251-bb924552009c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.246842 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.246803    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97979f66-3c4c-4410-b251-bb924552009c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.247035 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.246905    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94zg\" (UniqueName: \"kubernetes.io/projected/97979f66-3c4c-4410-b251-bb924552009c-kube-api-access-q94zg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.249405 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.249375    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97979f66-3c4c-4410-b251-bb924552009c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.258368 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.258333    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94zg\" (UniqueName: \"kubernetes.io/projected/97979f66-3c4c-4410-b251-bb924552009c-kube-api-access-q94zg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.347669 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.347578    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:28.631193 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:28.631162    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"]
Sep 05 03:11:28.634683 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:11:28.634644    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97979f66_3c4c_4410_b251_bb924552009c.slice/crio-e5be98342e9a1b5ce6268e7e75c1cf1a1f01d2fafa6c48f76862c864078032e2 WatchSource:0}: Error finding container e5be98342e9a1b5ce6268e7e75c1cf1a1f01d2fafa6c48f76862c864078032e2: Status 404 returned error can't find the container with id e5be98342e9a1b5ce6268e7e75c1cf1a1f01d2fafa6c48f76862c864078032e2
Sep 05 03:11:29.552939 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:29.552896    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" event={"ID":"97979f66-3c4c-4410-b251-bb924552009c","Type":"ContainerStarted","Data":"46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7"}
Sep 05 03:11:29.553377 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:29.552964    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" event={"ID":"97979f66-3c4c-4410-b251-bb924552009c","Type":"ContainerStarted","Data":"e5be98342e9a1b5ce6268e7e75c1cf1a1f01d2fafa6c48f76862c864078032e2"}
Sep 05 03:11:29.553377 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:29.553072    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:11:29.571341 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:11:29.571292    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podStartSLOduration=1.571273983 podStartE2EDuration="1.571273983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:11:29.570274127 +0000 UTC m=+3199.769637120" watchObservedRunningTime="2026-09-05 03:11:29.571273983 +0000 UTC m=+3199.770636975"
Sep 05 03:12:00.559382 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:12:00.559349    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:12:34.745146 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:12:34.745109    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"]
Sep 05 03:12:41.226427 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:12:41.226388    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:12:49.766953 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:12:49.766912    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main" containerID="cri-o://94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866" gracePeriod=45
Sep 05 03:12:51.226387 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:12:51.226346    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:12:59.625965 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:12:59.625925    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"]
Sep 05 03:13:00.559009 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:00.558969    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:01.226120 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:01.226081    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:01.226497 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:01.226178    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:13:10.559545 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:10.559511    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:11.225555 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:11.225517    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:14.645706 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:14.645645    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main" containerID="cri-o://46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7" gracePeriod=45
Sep 05 03:13:20.012046 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.012022    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:13:20.073122 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.073093    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c50ede2-4f4d-4043-8f49-97abc25496c6-tls-certs\") pod \"1c50ede2-4f4d-4043-8f49-97abc25496c6\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") "
Sep 05 03:13:20.073311 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.073133    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhp2l\" (UniqueName: \"kubernetes.io/projected/1c50ede2-4f4d-4043-8f49-97abc25496c6-kube-api-access-lhp2l\") pod \"1c50ede2-4f4d-4043-8f49-97abc25496c6\" (UID: \"1c50ede2-4f4d-4043-8f49-97abc25496c6\") "
Sep 05 03:13:20.075356 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.075321    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c50ede2-4f4d-4043-8f49-97abc25496c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c50ede2-4f4d-4043-8f49-97abc25496c6" (UID: "1c50ede2-4f4d-4043-8f49-97abc25496c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:13:20.075460 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.075357    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c50ede2-4f4d-4043-8f49-97abc25496c6-kube-api-access-lhp2l" (OuterVolumeSpecName: "kube-api-access-lhp2l") pod "1c50ede2-4f4d-4043-8f49-97abc25496c6" (UID: "1c50ede2-4f4d-4043-8f49-97abc25496c6"). InnerVolumeSpecName "kube-api-access-lhp2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:13:20.090003 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.089979    2553 generic.go:356] "Generic (PLEG): container finished" podID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerID="94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866" exitCode=0
Sep 05 03:13:20.090118 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.090017    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"
Sep 05 03:13:20.090118 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.090036    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" event={"ID":"1c50ede2-4f4d-4043-8f49-97abc25496c6","Type":"ContainerDied","Data":"94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866"}
Sep 05 03:13:20.090118 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.090081    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r" event={"ID":"1c50ede2-4f4d-4043-8f49-97abc25496c6","Type":"ContainerDied","Data":"f9d2efde1094d3589b63a9a925cca2ed5038f2745f11c52c7e28ecf0a367fb26"}
Sep 05 03:13:20.090118 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.090105    2553 scope.go:117] "RemoveContainer" containerID="94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866"
Sep 05 03:13:20.098733 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.098718    2553 scope.go:117] "RemoveContainer" containerID="94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866"
Sep 05 03:13:20.099000 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:13:20.098980    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866\": container with ID starting with 94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866 not found: ID does not exist" containerID="94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866"
Sep 05 03:13:20.099068 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.099007    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866"} err="failed to get container status \"94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866\": rpc error: code = NotFound desc = could not find container \"94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866\": container with ID starting with 94c9e64a8aee49199560990847cf18ba1bafcd0208ec15518a4a9af3b8ca3866 not found: ID does not exist"
Sep 05 03:13:20.111075 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.111049    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"]
Sep 05 03:13:20.113082 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.113061    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scherbq6r"]
Sep 05 03:13:20.176451 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.176363    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c50ede2-4f4d-4043-8f49-97abc25496c6-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:13:20.176546 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.176451    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp2l\" (UniqueName: \"kubernetes.io/projected/1c50ede2-4f4d-4043-8f49-97abc25496c6-kube-api-access-lhp2l\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:13:20.393392 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.393301    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" path="/var/lib/kubelet/pods/1c50ede2-4f4d-4043-8f49-97abc25496c6/volumes"
Sep 05 03:13:20.559024 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.558992    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:20.559248 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:20.559082    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:13:30.559206 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:30.559166    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:33.164925 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.164800    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"]
Sep 05 03:13:33.165370 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.165304    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main"
Sep 05 03:13:33.165370 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.165329    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main"
Sep 05 03:13:33.165500 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.165407    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c50ede2-4f4d-4043-8f49-97abc25496c6" containerName="main"
Sep 05 03:13:33.168631 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.168607    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.171259 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.171241    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-fq5qp\""
Sep 05 03:13:33.172386 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.172366    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 05 03:13:33.172516 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.172395    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-jncd2\""
Sep 05 03:13:33.172516 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.172419    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 05 03:13:33.172516 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.172395    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 05 03:13:33.179356 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.179334    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"]
Sep 05 03:13:33.283274 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.283243    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgtlw\" (UniqueName: \"kubernetes.io/projected/3529bdf7-60c2-441d-bb6d-fa9b86023191-kube-api-access-lgtlw\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.283446 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.283302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3529bdf7-60c2-441d-bb6d-fa9b86023191-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.383944 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.383909    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3529bdf7-60c2-441d-bb6d-fa9b86023191-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.384085 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.384064    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgtlw\" (UniqueName: \"kubernetes.io/projected/3529bdf7-60c2-441d-bb6d-fa9b86023191-kube-api-access-lgtlw\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.386530 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.386502    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3529bdf7-60c2-441d-bb6d-fa9b86023191-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.392853 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.392834    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgtlw\" (UniqueName: \"kubernetes.io/projected/3529bdf7-60c2-441d-bb6d-fa9b86023191-kube-api-access-lgtlw\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.482516 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.482431    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:33.761829 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:33.761798    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"]
Sep 05 03:13:33.763476 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:13:33.763451    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3529bdf7_60c2_441d_bb6d_fa9b86023191.slice/crio-1c44bb3cdfd5aac607fc7393eafba67a73a3c686ccfa59ff8753fc671137b54b WatchSource:0}: Error finding container 1c44bb3cdfd5aac607fc7393eafba67a73a3c686ccfa59ff8753fc671137b54b: Status 404 returned error can't find the container with id 1c44bb3cdfd5aac607fc7393eafba67a73a3c686ccfa59ff8753fc671137b54b
Sep 05 03:13:34.157524 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:34.157490    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" event={"ID":"3529bdf7-60c2-441d-bb6d-fa9b86023191","Type":"ContainerStarted","Data":"b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865"}
Sep 05 03:13:34.157705 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:34.157544    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" event={"ID":"3529bdf7-60c2-441d-bb6d-fa9b86023191","Type":"ContainerStarted","Data":"1c44bb3cdfd5aac607fc7393eafba67a73a3c686ccfa59ff8753fc671137b54b"}
Sep 05 03:13:34.157705 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:34.157582    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:13:34.175922 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:34.175862    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" podStartSLOduration=1.175845774 podStartE2EDuration="1.175845774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:13:34.174003486 +0000 UTC m=+3324.373366471" watchObservedRunningTime="2026-09-05 03:13:34.175845774 +0000 UTC m=+3324.375208767"
Sep 05 03:13:40.559694 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:40.559659    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:13:44.894177 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:44.894153    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:13:44.975297 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:44.975223    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q94zg\" (UniqueName: \"kubernetes.io/projected/97979f66-3c4c-4410-b251-bb924552009c-kube-api-access-q94zg\") pod \"97979f66-3c4c-4410-b251-bb924552009c\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") "
Sep 05 03:13:44.975297 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:44.975268    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97979f66-3c4c-4410-b251-bb924552009c-tls-certs\") pod \"97979f66-3c4c-4410-b251-bb924552009c\" (UID: \"97979f66-3c4c-4410-b251-bb924552009c\") "
Sep 05 03:13:44.977317 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:44.977292    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97979f66-3c4c-4410-b251-bb924552009c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97979f66-3c4c-4410-b251-bb924552009c" (UID: "97979f66-3c4c-4410-b251-bb924552009c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:13:44.977417 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:44.977353    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97979f66-3c4c-4410-b251-bb924552009c-kube-api-access-q94zg" (OuterVolumeSpecName: "kube-api-access-q94zg") pod "97979f66-3c4c-4410-b251-bb924552009c" (UID: "97979f66-3c4c-4410-b251-bb924552009c"). InnerVolumeSpecName "kube-api-access-q94zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:13:45.076126 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.076099    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94zg\" (UniqueName: \"kubernetes.io/projected/97979f66-3c4c-4410-b251-bb924552009c-kube-api-access-q94zg\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:13:45.076228 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.076127    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97979f66-3c4c-4410-b251-bb924552009c-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:13:45.210390 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.210359    2553 generic.go:356] "Generic (PLEG): container finished" podID="97979f66-3c4c-4410-b251-bb924552009c" containerID="46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7" exitCode=0
Sep 05 03:13:45.210495 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.210401    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"
Sep 05 03:13:45.210495 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.210447    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" event={"ID":"97979f66-3c4c-4410-b251-bb924552009c","Type":"ContainerDied","Data":"46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7"}
Sep 05 03:13:45.210594 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.210497    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn" event={"ID":"97979f66-3c4c-4410-b251-bb924552009c","Type":"ContainerDied","Data":"e5be98342e9a1b5ce6268e7e75c1cf1a1f01d2fafa6c48f76862c864078032e2"}
Sep 05 03:13:45.210594 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.210522    2553 scope.go:117] "RemoveContainer" containerID="46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7"
Sep 05 03:13:45.221203 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.221176    2553 scope.go:117] "RemoveContainer" containerID="46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7"
Sep 05 03:13:45.221736 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:13:45.221708    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7\": container with ID starting with 46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7 not found: ID does not exist" containerID="46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7"
Sep 05 03:13:45.221823 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.221747    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7"} err="failed to get container status \"46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7\": rpc error: code = NotFound desc = could not find container \"46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7\": container with ID starting with 46f85801d10b7444debd51ff57663c5351c6a77aa3b6de253c2a2886cf5dead7 not found: ID does not exist"
Sep 05 03:13:45.234420 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.234371    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"]
Sep 05 03:13:45.238186 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:45.238166    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schespxkn"]
Sep 05 03:13:46.393450 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:13:46.393413    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97979f66-3c4c-4410-b251-bb924552009c" path="/var/lib/kubelet/pods/97979f66-3c4c-4410-b251-bb924552009c/volumes"
Sep 05 03:14:02.340218 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.340182    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"]
Sep 05 03:14:02.340689 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.340552    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main"
Sep 05 03:14:02.340689 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.340580    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main"
Sep 05 03:14:02.340689 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.340648    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97979f66-3c4c-4410-b251-bb924552009c" containerName="main"
Sep 05 03:14:02.343157 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.343135    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.346884 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.346850    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 05 03:14:02.347096 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.347068    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 05 03:14:02.347222 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.346915    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-xh7w2\""
Sep 05 03:14:02.347222 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.346950    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 05 03:14:02.347222 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.347021    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-zdhn6\""
Sep 05 03:14:02.351972 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.351951    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"]
Sep 05 03:14:02.425746 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.425721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a794b-518f-44d2-a27f-37d1b8d959ca-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.425888 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.425752    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jkh\" (UniqueName: \"kubernetes.io/projected/e47a794b-518f-44d2-a27f-37d1b8d959ca-kube-api-access-p2jkh\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.526921 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.526895    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a794b-518f-44d2-a27f-37d1b8d959ca-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.527057 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.526931    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jkh\" (UniqueName: \"kubernetes.io/projected/e47a794b-518f-44d2-a27f-37d1b8d959ca-kube-api-access-p2jkh\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.529351 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.529326    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a794b-518f-44d2-a27f-37d1b8d959ca-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.534543 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.534523    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jkh\" (UniqueName: \"kubernetes.io/projected/e47a794b-518f-44d2-a27f-37d1b8d959ca-kube-api-access-p2jkh\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.654590 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.654513    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:02.940803 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:02.940771    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"]
Sep 05 03:14:02.942611 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:14:02.942585    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47a794b_518f_44d2_a27f_37d1b8d959ca.slice/crio-1b17b0d2356265a94de972221da9c48e66b470827dd869ff1cabd8e779dc80f1 WatchSource:0}: Error finding container 1b17b0d2356265a94de972221da9c48e66b470827dd869ff1cabd8e779dc80f1: Status 404 returned error can't find the container with id 1b17b0d2356265a94de972221da9c48e66b470827dd869ff1cabd8e779dc80f1
Sep 05 03:14:03.308898 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:03.308860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" event={"ID":"e47a794b-518f-44d2-a27f-37d1b8d959ca","Type":"ContainerStarted","Data":"5168321d6b75e7b8665df36d331afd3da56e31e58a08d0af79bae378b60d0aa2"}
Sep 05 03:14:03.309097 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:03.308945    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" event={"ID":"e47a794b-518f-44d2-a27f-37d1b8d959ca","Type":"ContainerStarted","Data":"1b17b0d2356265a94de972221da9c48e66b470827dd869ff1cabd8e779dc80f1"}
Sep 05 03:14:03.309097 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:03.308971    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:03.329350 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:03.329291    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" podStartSLOduration=1.329272645 podStartE2EDuration="1.329272645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:14:03.32875273 +0000 UTC m=+3353.528115724" watchObservedRunningTime="2026-09-05 03:14:03.329272645 +0000 UTC m=+3353.528635638"
Sep 05 03:14:05.164787 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:05.164754    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:14:05.211213 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:05.211179    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"]
Sep 05 03:14:20.338149 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:20.338081    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerName="main" containerID="cri-o://5168321d6b75e7b8665df36d331afd3da56e31e58a08d0af79bae378b60d0aa2" gracePeriod=45
Sep 05 03:14:35.321350 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:35.321310    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:14:37.257259 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.257227    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"]
Sep 05 03:14:37.261114 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.261093    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.263706 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.263680    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 05 03:14:37.263826 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.263715    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 05 03:14:37.264856 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.264838    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-rpdns\""
Sep 05 03:14:37.264856 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.264845    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 05 03:14:37.264991 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.264863    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-2f47l\""
Sep 05 03:14:37.268895 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.268877    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"]
Sep 05 03:14:37.407160 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.407123    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4f4\" (UniqueName: \"kubernetes.io/projected/8253ff67-c300-42ea-8056-c852e71646b9-kube-api-access-vv4f4\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.407326 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.407205    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8253ff67-c300-42ea-8056-c852e71646b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.508018 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.507942    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8253ff67-c300-42ea-8056-c852e71646b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.508018 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.507992    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4f4\" (UniqueName: \"kubernetes.io/projected/8253ff67-c300-42ea-8056-c852e71646b9-kube-api-access-vv4f4\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.510374 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.510343    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8253ff67-c300-42ea-8056-c852e71646b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.515913 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.515888    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4f4\" (UniqueName: \"kubernetes.io/projected/8253ff67-c300-42ea-8056-c852e71646b9-kube-api-access-vv4f4\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.571838 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.571809    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:37.845298 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:37.844107    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"]
Sep 05 03:14:38.478492 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:38.478459    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" event={"ID":"8253ff67-c300-42ea-8056-c852e71646b9","Type":"ContainerStarted","Data":"16def84d0e2fc2b695b6edeb4eeeb3a9b4840d76c037fd72c5872a735b95fbe2"}
Sep 05 03:14:38.478926 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:38.478510    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" event={"ID":"8253ff67-c300-42ea-8056-c852e71646b9","Type":"ContainerStarted","Data":"d42ce2e79de872d0a4316f0ee825cc16e93c0550d58031d33800ec0dfce39f0d"}
Sep 05 03:14:38.478926 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:38.478592    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:14:38.494575 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:38.494520    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" podStartSLOduration=1.494504657 podStartE2EDuration="1.494504657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:14:38.493707038 +0000 UTC m=+3388.693070032" watchObservedRunningTime="2026-09-05 03:14:38.494504657 +0000 UTC m=+3388.693867654"
Sep 05 03:14:45.320172 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:45.320135    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:14:50.535534 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.535503    2553 generic.go:356] "Generic (PLEG): container finished" podID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerID="5168321d6b75e7b8665df36d331afd3da56e31e58a08d0af79bae378b60d0aa2" exitCode=0
Sep 05 03:14:50.535928 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.535589    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" event={"ID":"e47a794b-518f-44d2-a27f-37d1b8d959ca","Type":"ContainerDied","Data":"5168321d6b75e7b8665df36d331afd3da56e31e58a08d0af79bae378b60d0aa2"}
Sep 05 03:14:50.586932 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.586908    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:50.721280 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.721191    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a794b-518f-44d2-a27f-37d1b8d959ca-tls-certs\") pod \"e47a794b-518f-44d2-a27f-37d1b8d959ca\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") "
Sep 05 03:14:50.721280 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.721260    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2jkh\" (UniqueName: \"kubernetes.io/projected/e47a794b-518f-44d2-a27f-37d1b8d959ca-kube-api-access-p2jkh\") pod \"e47a794b-518f-44d2-a27f-37d1b8d959ca\" (UID: \"e47a794b-518f-44d2-a27f-37d1b8d959ca\") "
Sep 05 03:14:50.723413 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.723381    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47a794b-518f-44d2-a27f-37d1b8d959ca-kube-api-access-p2jkh" (OuterVolumeSpecName: "kube-api-access-p2jkh") pod "e47a794b-518f-44d2-a27f-37d1b8d959ca" (UID: "e47a794b-518f-44d2-a27f-37d1b8d959ca"). InnerVolumeSpecName "kube-api-access-p2jkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:14:50.723532 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.723420    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47a794b-518f-44d2-a27f-37d1b8d959ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e47a794b-518f-44d2-a27f-37d1b8d959ca" (UID: "e47a794b-518f-44d2-a27f-37d1b8d959ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:14:50.822132 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.822101    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jkh\" (UniqueName: \"kubernetes.io/projected/e47a794b-518f-44d2-a27f-37d1b8d959ca-kube-api-access-p2jkh\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:14:50.822132 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:50.822127    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a794b-518f-44d2-a27f-37d1b8d959ca-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:14:51.541035 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:51.541009    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"
Sep 05 03:14:51.541035 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:51.541016    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq" event={"ID":"e47a794b-518f-44d2-a27f-37d1b8d959ca","Type":"ContainerDied","Data":"1b17b0d2356265a94de972221da9c48e66b470827dd869ff1cabd8e779dc80f1"}
Sep 05 03:14:51.541546 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:51.541072    2553 scope.go:117] "RemoveContainer" containerID="5168321d6b75e7b8665df36d331afd3da56e31e58a08d0af79bae378b60d0aa2"
Sep 05 03:14:51.562440 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:51.562413    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"]
Sep 05 03:14:51.566206 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:51.566182    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-klhxq"]
Sep 05 03:14:52.393737 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:14:52.393704    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" path="/var/lib/kubelet/pods/e47a794b-518f-44d2-a27f-37d1b8d959ca/volumes"
Sep 05 03:15:09.484664 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:15:09.484586    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:15:38.329439 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:15:38.329404    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"]
Sep 05 03:15:45.164462 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:15:45.164420    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:15:53.358098 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:15:53.358059    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main" containerID="cri-o://b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865" gracePeriod=45
Sep 05 03:15:55.164326 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:15:55.164290    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:16:05.165005 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:05.164966    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:16:05.165491 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:05.165081    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:16:15.164921 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:15.164883    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:16:23.619950 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.619881    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:16:23.669916 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.669875    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgtlw\" (UniqueName: \"kubernetes.io/projected/3529bdf7-60c2-441d-bb6d-fa9b86023191-kube-api-access-lgtlw\") pod \"3529bdf7-60c2-441d-bb6d-fa9b86023191\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") "
Sep 05 03:16:23.670113 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.669928    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3529bdf7-60c2-441d-bb6d-fa9b86023191-tls-certs\") pod \"3529bdf7-60c2-441d-bb6d-fa9b86023191\" (UID: \"3529bdf7-60c2-441d-bb6d-fa9b86023191\") "
Sep 05 03:16:23.672207 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.672180    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3529bdf7-60c2-441d-bb6d-fa9b86023191-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3529bdf7-60c2-441d-bb6d-fa9b86023191" (UID: "3529bdf7-60c2-441d-bb6d-fa9b86023191"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:16:23.672277 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.672218    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3529bdf7-60c2-441d-bb6d-fa9b86023191-kube-api-access-lgtlw" (OuterVolumeSpecName: "kube-api-access-lgtlw") pod "3529bdf7-60c2-441d-bb6d-fa9b86023191" (UID: "3529bdf7-60c2-441d-bb6d-fa9b86023191"). InnerVolumeSpecName "kube-api-access-lgtlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:16:23.771046 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.771003    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgtlw\" (UniqueName: \"kubernetes.io/projected/3529bdf7-60c2-441d-bb6d-fa9b86023191-kube-api-access-lgtlw\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:16:23.771046 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.771040    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3529bdf7-60c2-441d-bb6d-fa9b86023191-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:16:23.978217 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.978122    2553 generic.go:356] "Generic (PLEG): container finished" podID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerID="b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865" exitCode=0
Sep 05 03:16:23.978217 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.978171    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"
Sep 05 03:16:23.978453 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.978207    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" event={"ID":"3529bdf7-60c2-441d-bb6d-fa9b86023191","Type":"ContainerDied","Data":"b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865"}
Sep 05 03:16:23.978453 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.978267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5" event={"ID":"3529bdf7-60c2-441d-bb6d-fa9b86023191","Type":"ContainerDied","Data":"1c44bb3cdfd5aac607fc7393eafba67a73a3c686ccfa59ff8753fc671137b54b"}
Sep 05 03:16:23.978453 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.978290    2553 scope.go:117] "RemoveContainer" containerID="b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865"
Sep 05 03:16:23.987675 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.987652    2553 scope.go:117] "RemoveContainer" containerID="b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865"
Sep 05 03:16:23.987958 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:16:23.987935    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865\": container with ID starting with b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865 not found: ID does not exist" containerID="b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865"
Sep 05 03:16:23.988034 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.987968    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865"} err="failed to get container status \"b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865\": rpc error: code = NotFound desc = could not find container \"b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865\": container with ID starting with b6fc07b4638d4e48197b40a917ab44b0e51de5758a7c560c1978d3d6097d5865 not found: ID does not exist"
Sep 05 03:16:23.999793 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:23.999769    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"]
Sep 05 03:16:24.002095 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:24.002075    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58sgnl5"]
Sep 05 03:16:24.393757 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:16:24.393721    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" path="/var/lib/kubelet/pods/3529bdf7-60c2-441d-bb6d-fa9b86023191/volumes"
Sep 05 03:17:01.476272 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476238    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"]
Sep 05 03:17:01.476902 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476666    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerName="main"
Sep 05 03:17:01.476902 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476687    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerName="main"
Sep 05 03:17:01.476902 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476702    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main"
Sep 05 03:17:01.476902 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476710    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main"
Sep 05 03:17:01.476902 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476817    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47a794b-518f-44d2-a27f-37d1b8d959ca" containerName="main"
Sep 05 03:17:01.476902 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.476833    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3529bdf7-60c2-441d-bb6d-fa9b86023191" containerName="main"
Sep 05 03:17:01.479972 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.479944    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.483717 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.483454    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-lmnqm\""
Sep 05 03:17:01.483717 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.483474    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 05 03:17:01.483717 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.483477    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 05 03:17:01.483717 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.483536    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 05 03:17:01.483717 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.483464    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-fmk7q\""
Sep 05 03:17:01.490184 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.490163    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"]
Sep 05 03:17:01.571042 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.571014    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5782b292-ff6f-4224-9224-bc3d748d1631-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.571184 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.571115    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp872\" (UniqueName: \"kubernetes.io/projected/5782b292-ff6f-4224-9224-bc3d748d1631-kube-api-access-mp872\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.672249 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.672222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5782b292-ff6f-4224-9224-bc3d748d1631-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.672353 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.672288    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp872\" (UniqueName: \"kubernetes.io/projected/5782b292-ff6f-4224-9224-bc3d748d1631-kube-api-access-mp872\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.674779 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.674746    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5782b292-ff6f-4224-9224-bc3d748d1631-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.680554 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.680529    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp872\" (UniqueName: \"kubernetes.io/projected/5782b292-ff6f-4224-9224-bc3d748d1631-kube-api-access-mp872\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:01.792177 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:01.792148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:02.064153 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:02.064117    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"]
Sep 05 03:17:02.066687 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:17:02.066660    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5782b292_ff6f_4224_9224_bc3d748d1631.slice/crio-7b41c3803594a6f9b824c2dc40aeecc7c1b834a943b16c2d69268ec9937766e0 WatchSource:0}: Error finding container 7b41c3803594a6f9b824c2dc40aeecc7c1b834a943b16c2d69268ec9937766e0: Status 404 returned error can't find the container with id 7b41c3803594a6f9b824c2dc40aeecc7c1b834a943b16c2d69268ec9937766e0
Sep 05 03:17:02.156227 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:02.156188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" event={"ID":"5782b292-ff6f-4224-9224-bc3d748d1631","Type":"ContainerStarted","Data":"080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e"}
Sep 05 03:17:02.156384 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:02.156259    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" event={"ID":"5782b292-ff6f-4224-9224-bc3d748d1631","Type":"ContainerStarted","Data":"7b41c3803594a6f9b824c2dc40aeecc7c1b834a943b16c2d69268ec9937766e0"}
Sep 05 03:17:02.156546 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:02.156514    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:02.173184 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:02.173144    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podStartSLOduration=1.173131411 podStartE2EDuration="1.173131411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:17:02.172001348 +0000 UTC m=+3532.371364362" watchObservedRunningTime="2026-09-05 03:17:02.173131411 +0000 UTC m=+3532.372494403"
Sep 05 03:17:12.265632 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:12.265599    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"]
Sep 05 03:17:19.485187 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:19.485140    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:17:27.286015 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:27.285970    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main" containerID="cri-o://16def84d0e2fc2b695b6edeb4eeeb3a9b4840d76c037fd72c5872a735b95fbe2" gracePeriod=45
Sep 05 03:17:29.484965 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:29.484921    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:17:33.162556 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:33.162522    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:17:39.484966 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:39.484924    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:17:39.485428 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:39.485029    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:17:49.485627 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:49.485585    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:17:57.423905 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.423870    2553 generic.go:356] "Generic (PLEG): container finished" podID="8253ff67-c300-42ea-8056-c852e71646b9" containerID="16def84d0e2fc2b695b6edeb4eeeb3a9b4840d76c037fd72c5872a735b95fbe2" exitCode=0
Sep 05 03:17:57.424328 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.423941    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" event={"ID":"8253ff67-c300-42ea-8056-c852e71646b9","Type":"ContainerDied","Data":"16def84d0e2fc2b695b6edeb4eeeb3a9b4840d76c037fd72c5872a735b95fbe2"}
Sep 05 03:17:57.527733 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.527703    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:17:57.638469 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.638388    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8253ff67-c300-42ea-8056-c852e71646b9-tls-certs\") pod \"8253ff67-c300-42ea-8056-c852e71646b9\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") "
Sep 05 03:17:57.638469 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.638440    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv4f4\" (UniqueName: \"kubernetes.io/projected/8253ff67-c300-42ea-8056-c852e71646b9-kube-api-access-vv4f4\") pod \"8253ff67-c300-42ea-8056-c852e71646b9\" (UID: \"8253ff67-c300-42ea-8056-c852e71646b9\") "
Sep 05 03:17:57.640467 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.640440    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8253ff67-c300-42ea-8056-c852e71646b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8253ff67-c300-42ea-8056-c852e71646b9" (UID: "8253ff67-c300-42ea-8056-c852e71646b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:17:57.640620 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.640498    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8253ff67-c300-42ea-8056-c852e71646b9-kube-api-access-vv4f4" (OuterVolumeSpecName: "kube-api-access-vv4f4") pod "8253ff67-c300-42ea-8056-c852e71646b9" (UID: "8253ff67-c300-42ea-8056-c852e71646b9"). InnerVolumeSpecName "kube-api-access-vv4f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:17:57.739893 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.739867    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8253ff67-c300-42ea-8056-c852e71646b9-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:17:57.739893 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:57.739891    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv4f4\" (UniqueName: \"kubernetes.io/projected/8253ff67-c300-42ea-8056-c852e71646b9-kube-api-access-vv4f4\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:17:58.430070 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:58.430033    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k" event={"ID":"8253ff67-c300-42ea-8056-c852e71646b9","Type":"ContainerDied","Data":"d42ce2e79de872d0a4316f0ee825cc16e93c0550d58031d33800ec0dfce39f0d"}
Sep 05 03:17:58.430425 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:58.430096    2553 scope.go:117] "RemoveContainer" containerID="16def84d0e2fc2b695b6edeb4eeeb3a9b4840d76c037fd72c5872a735b95fbe2"
Sep 05 03:17:58.430425 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:58.430045    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"
Sep 05 03:17:58.445753 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:58.445729    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"]
Sep 05 03:17:58.449316 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:17:58.449290    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-9944k"]
Sep 05 03:18:00.392654 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:18:00.392614    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8253ff67-c300-42ea-8056-c852e71646b9" path="/var/lib/kubelet/pods/8253ff67-c300-42ea-8056-c852e71646b9/volumes"
Sep 05 03:19:09.219600 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:09.219543    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"]
Sep 05 03:19:13.162007 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:13.161966    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:19:23.162331 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:23.162282    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:19:24.240406 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:24.240344    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main" containerID="cri-o://080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e" gracePeriod=45
Sep 05 03:19:33.162475 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:33.162385    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:19:33.162896 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:33.162488    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:19:43.162246 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:43.162204    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:19:53.162416 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:53.162378    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:19:54.481395 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.481368    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:19:54.560582 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.560539    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5782b292-ff6f-4224-9224-bc3d748d1631-tls-certs\") pod \"5782b292-ff6f-4224-9224-bc3d748d1631\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") "
Sep 05 03:19:54.560735 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.560619    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp872\" (UniqueName: \"kubernetes.io/projected/5782b292-ff6f-4224-9224-bc3d748d1631-kube-api-access-mp872\") pod \"5782b292-ff6f-4224-9224-bc3d748d1631\" (UID: \"5782b292-ff6f-4224-9224-bc3d748d1631\") "
Sep 05 03:19:54.562724 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.562693    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5782b292-ff6f-4224-9224-bc3d748d1631-kube-api-access-mp872" (OuterVolumeSpecName: "kube-api-access-mp872") pod "5782b292-ff6f-4224-9224-bc3d748d1631" (UID: "5782b292-ff6f-4224-9224-bc3d748d1631"). InnerVolumeSpecName "kube-api-access-mp872". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:19:54.562829 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.562719    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5782b292-ff6f-4224-9224-bc3d748d1631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5782b292-ff6f-4224-9224-bc3d748d1631" (UID: "5782b292-ff6f-4224-9224-bc3d748d1631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:19:54.662036 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.662011    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5782b292-ff6f-4224-9224-bc3d748d1631-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:19:54.662036 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.662037    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp872\" (UniqueName: \"kubernetes.io/projected/5782b292-ff6f-4224-9224-bc3d748d1631-kube-api-access-mp872\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:19:54.974778 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.974692    2553 generic.go:356] "Generic (PLEG): container finished" podID="5782b292-ff6f-4224-9224-bc3d748d1631" containerID="080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e" exitCode=0
Sep 05 03:19:54.974778 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.974748    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"
Sep 05 03:19:54.975008 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.974779    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" event={"ID":"5782b292-ff6f-4224-9224-bc3d748d1631","Type":"ContainerDied","Data":"080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e"}
Sep 05 03:19:54.975008 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.974849    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn" event={"ID":"5782b292-ff6f-4224-9224-bc3d748d1631","Type":"ContainerDied","Data":"7b41c3803594a6f9b824c2dc40aeecc7c1b834a943b16c2d69268ec9937766e0"}
Sep 05 03:19:54.975008 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.974880    2553 scope.go:117] "RemoveContainer" containerID="080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e"
Sep 05 03:19:54.983840 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.983819    2553 scope.go:117] "RemoveContainer" containerID="080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e"
Sep 05 03:19:54.984121 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:19:54.984104    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e\": container with ID starting with 080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e not found: ID does not exist" containerID="080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e"
Sep 05 03:19:54.984169 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.984129    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e"} err="failed to get container status \"080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e\": rpc error: code = NotFound desc = could not find container \"080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e\": container with ID starting with 080029ce3c3f9de1f2784df1fd289028e9ae92da1ead24e2be2d139b1d8ea02e not found: ID does not exist"
Sep 05 03:19:54.995095 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.995070    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"]
Sep 05 03:19:54.998634 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:54.998611    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-8wsjn"]
Sep 05 03:19:56.393057 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:19:56.393020    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" path="/var/lib/kubelet/pods/5782b292-ff6f-4224-9224-bc3d748d1631/volumes"
Sep 05 03:20:07.481431 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481392    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"]
Sep 05 03:20:07.481849 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481754    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main"
Sep 05 03:20:07.481849 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481767    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main"
Sep 05 03:20:07.481849 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481799    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main"
Sep 05 03:20:07.481849 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481804    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main"
Sep 05 03:20:07.481987 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481854    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5782b292-ff6f-4224-9224-bc3d748d1631" containerName="main"
Sep 05 03:20:07.481987 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.481866    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8253ff67-c300-42ea-8056-c852e71646b9" containerName="main"
Sep 05 03:20:07.485013 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.484995    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.488526 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.488497    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-hsdmz\""
Sep 05 03:20:07.488688 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.488543    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 05 03:20:07.488859 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.488843    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 05 03:20:07.488905 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.488865    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 05 03:20:07.488951 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.488844    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-dtt9d\""
Sep 05 03:20:07.493661 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.493638    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"]
Sep 05 03:20:07.565948 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.565919    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdqt\" (UniqueName: \"kubernetes.io/projected/930f672a-05a0-44b2-936a-41365dcb2dec-kube-api-access-rtdqt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.566087 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.566022    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930f672a-05a0-44b2-936a-41365dcb2dec-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.667159 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.667131    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930f672a-05a0-44b2-936a-41365dcb2dec-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.667341 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.667175    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdqt\" (UniqueName: \"kubernetes.io/projected/930f672a-05a0-44b2-936a-41365dcb2dec-kube-api-access-rtdqt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.669477 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.669447    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930f672a-05a0-44b2-936a-41365dcb2dec-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.675195 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.675173    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdqt\" (UniqueName: \"kubernetes.io/projected/930f672a-05a0-44b2-936a-41365dcb2dec-kube-api-access-rtdqt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:07.797343 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:07.797308    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:08.076097 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:08.076009    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"]
Sep 05 03:20:08.079484 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:20:08.079455    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930f672a_05a0_44b2_936a_41365dcb2dec.slice/crio-b00a5212ac05cd13817043dbffd971175cab9a191368ddbf33056ff3a980c4d8 WatchSource:0}: Error finding container b00a5212ac05cd13817043dbffd971175cab9a191368ddbf33056ff3a980c4d8: Status 404 returned error can't find the container with id b00a5212ac05cd13817043dbffd971175cab9a191368ddbf33056ff3a980c4d8
Sep 05 03:20:09.044828 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:09.044793    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" event={"ID":"930f672a-05a0-44b2-936a-41365dcb2dec","Type":"ContainerStarted","Data":"d9bb10edd3d7ae952d7478d4472a98b7564eff64d26a0f6241b90eaf43866b00"}
Sep 05 03:20:09.045234 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:09.044850    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" event={"ID":"930f672a-05a0-44b2-936a-41365dcb2dec","Type":"ContainerStarted","Data":"b00a5212ac05cd13817043dbffd971175cab9a191368ddbf33056ff3a980c4d8"}
Sep 05 03:20:09.045234 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:09.044887    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:20:09.063468 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:09.063421    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" podStartSLOduration=2.063408798 podStartE2EDuration="2.063408798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:20:09.061507992 +0000 UTC m=+3719.260871002" watchObservedRunningTime="2026-09-05 03:20:09.063408798 +0000 UTC m=+3719.262771790"
Sep 05 03:20:40.052003 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:20:40.051969    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:22:30.788126 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:22:30.788077    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"]
Sep 05 03:22:40.051987 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:22:40.051896    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:22:45.809977 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:22:45.809914    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main" containerID="cri-o://d9bb10edd3d7ae952d7478d4472a98b7564eff64d26a0f6241b90eaf43866b00" gracePeriod=45
Sep 05 03:22:50.051400 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:22:50.051360    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:23:00.051146 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:00.051107    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:23:00.051626 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:00.051206    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:23:10.052049 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:10.052004    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:23:15.952495 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:15.952461    2553 generic.go:356] "Generic (PLEG): container finished" podID="930f672a-05a0-44b2-936a-41365dcb2dec" containerID="d9bb10edd3d7ae952d7478d4472a98b7564eff64d26a0f6241b90eaf43866b00" exitCode=0
Sep 05 03:23:15.952925 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:15.952537    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" event={"ID":"930f672a-05a0-44b2-936a-41365dcb2dec","Type":"ContainerDied","Data":"d9bb10edd3d7ae952d7478d4472a98b7564eff64d26a0f6241b90eaf43866b00"}
Sep 05 03:23:16.047428 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.047406    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:23:16.106804 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.106731    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtdqt\" (UniqueName: \"kubernetes.io/projected/930f672a-05a0-44b2-936a-41365dcb2dec-kube-api-access-rtdqt\") pod \"930f672a-05a0-44b2-936a-41365dcb2dec\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") "
Sep 05 03:23:16.106804 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.106768    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930f672a-05a0-44b2-936a-41365dcb2dec-tls-certs\") pod \"930f672a-05a0-44b2-936a-41365dcb2dec\" (UID: \"930f672a-05a0-44b2-936a-41365dcb2dec\") "
Sep 05 03:23:16.108950 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.108920    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930f672a-05a0-44b2-936a-41365dcb2dec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "930f672a-05a0-44b2-936a-41365dcb2dec" (UID: "930f672a-05a0-44b2-936a-41365dcb2dec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:23:16.109067 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.109000    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930f672a-05a0-44b2-936a-41365dcb2dec-kube-api-access-rtdqt" (OuterVolumeSpecName: "kube-api-access-rtdqt") pod "930f672a-05a0-44b2-936a-41365dcb2dec" (UID: "930f672a-05a0-44b2-936a-41365dcb2dec"). InnerVolumeSpecName "kube-api-access-rtdqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:23:16.207899 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.207862    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdqt\" (UniqueName: \"kubernetes.io/projected/930f672a-05a0-44b2-936a-41365dcb2dec-kube-api-access-rtdqt\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:23:16.207899 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.207895    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930f672a-05a0-44b2-936a-41365dcb2dec-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:23:16.959139 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.959103    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"
Sep 05 03:23:16.959624 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.959136    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr" event={"ID":"930f672a-05a0-44b2-936a-41365dcb2dec","Type":"ContainerDied","Data":"b00a5212ac05cd13817043dbffd971175cab9a191368ddbf33056ff3a980c4d8"}
Sep 05 03:23:16.959624 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.959200    2553 scope.go:117] "RemoveContainer" containerID="d9bb10edd3d7ae952d7478d4472a98b7564eff64d26a0f6241b90eaf43866b00"
Sep 05 03:23:16.977348 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.977318    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"]
Sep 05 03:23:16.982941 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:16.982917    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schehptwr"]
Sep 05 03:23:18.392839 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:18.392805    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" path="/var/lib/kubelet/pods/930f672a-05a0-44b2-936a-41365dcb2dec/volumes"
Sep 05 03:23:36.431623 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.431582    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"]
Sep 05 03:23:36.432037 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.431991    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main"
Sep 05 03:23:36.432037 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.432005    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main"
Sep 05 03:23:36.432119 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.432084    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="930f672a-05a0-44b2-936a-41365dcb2dec" containerName="main"
Sep 05 03:23:36.435165 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.435146    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.437888 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.437864    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-b2587\""
Sep 05 03:23:36.441009 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.440987    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-9xmjg\""
Sep 05 03:23:36.442073 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.442047    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 05 03:23:36.442184 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.442095    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 05 03:23:36.442339 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.442320    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 05 03:23:36.448278 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.448253    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"]
Sep 05 03:23:36.496109 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.496076    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpmm\" (UniqueName: \"kubernetes.io/projected/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-kube-api-access-qmpmm\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.496283 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.496136    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.597607 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.597577    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpmm\" (UniqueName: \"kubernetes.io/projected/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-kube-api-access-qmpmm\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.597772 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.597612    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.600050 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.600018    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.608836 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.608813    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpmm\" (UniqueName: \"kubernetes.io/projected/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-kube-api-access-qmpmm\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:36.745517 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:36.745432    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:37.019245 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:37.019032    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"]
Sep 05 03:23:37.021819 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:23:37.021784    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e9be74_5b3c_4d4d_808d_145614d8ae1e.slice/crio-e1846c45da09c61d2221b33598c9b542f658b8a2253a518c89724b52aa7d6a1d WatchSource:0}: Error finding container e1846c45da09c61d2221b33598c9b542f658b8a2253a518c89724b52aa7d6a1d: Status 404 returned error can't find the container with id e1846c45da09c61d2221b33598c9b542f658b8a2253a518c89724b52aa7d6a1d
Sep 05 03:23:37.058417 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:37.058389    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" event={"ID":"d6e9be74-5b3c-4d4d-808d-145614d8ae1e","Type":"ContainerStarted","Data":"e1846c45da09c61d2221b33598c9b542f658b8a2253a518c89724b52aa7d6a1d"}
Sep 05 03:23:38.065405 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:38.065359    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" event={"ID":"d6e9be74-5b3c-4d4d-808d-145614d8ae1e","Type":"ContainerStarted","Data":"7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12"}
Sep 05 03:23:38.065855 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:38.065430    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:23:38.083326 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:23:38.083276    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podStartSLOduration=2.083261742 podStartE2EDuration="2.083261742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:23:38.081322863 +0000 UTC m=+3928.280685871" watchObservedRunningTime="2026-09-05 03:23:38.083261742 +0000 UTC m=+3928.282624735"
Sep 05 03:24:09.073584 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:24:09.073490    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:26:15.022394 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.022350    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"]
Sep 05 03:26:15.026310 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.026285    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.029941 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.029898    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 05 03:26:15.029941 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.029919    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 05 03:26:15.030131 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.029972    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 05 03:26:15.030131 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.029926    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-6g6g9\""
Sep 05 03:26:15.030131 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.029919    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-p24j7\""
Sep 05 03:26:15.037397 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.037375    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"]
Sep 05 03:26:15.106096 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.106053    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.106285 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.106108    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwrb\" (UniqueName: \"kubernetes.io/projected/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-kube-api-access-btwrb\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.207467 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.207429    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.207693 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.207477    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwrb\" (UniqueName: \"kubernetes.io/projected/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-kube-api-access-btwrb\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.210029 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.209999    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.215268 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.215246    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwrb\" (UniqueName: \"kubernetes.io/projected/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-kube-api-access-btwrb\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.336716 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.336676    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.615184 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.615157    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"]
Sep 05 03:26:15.617845 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:26:15.617816    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ce8ee4_a3b6_4bfc_bd3b_c23a0e978f37.slice/crio-c370a18a3d8f9fc692571913e8bdbb56b46256f34baeee438b2ad014b858501b WatchSource:0}: Error finding container c370a18a3d8f9fc692571913e8bdbb56b46256f34baeee438b2ad014b858501b: Status 404 returned error can't find the container with id c370a18a3d8f9fc692571913e8bdbb56b46256f34baeee438b2ad014b858501b
Sep 05 03:26:15.807413 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.807371    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" event={"ID":"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37","Type":"ContainerStarted","Data":"3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6"}
Sep 05 03:26:15.807634 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.807438    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" event={"ID":"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37","Type":"ContainerStarted","Data":"c370a18a3d8f9fc692571913e8bdbb56b46256f34baeee438b2ad014b858501b"}
Sep 05 03:26:15.807634 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.807534    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:15.823820 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:15.823765    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" podStartSLOduration=0.823750124 podStartE2EDuration="823.750124ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:26:15.822730461 +0000 UTC m=+4086.022093468" watchObservedRunningTime="2026-09-05 03:26:15.823750124 +0000 UTC m=+4086.023113117"
Sep 05 03:26:34.723338 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:34.723303    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"]
Sep 05 03:26:39.073395 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:39.073358    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:26:46.814283 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:46.814253    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:26:49.073992 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:49.073956    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:26:49.743119 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:49.743073    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main" containerID="cri-o://7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12" gracePeriod=45
Sep 05 03:26:59.073271 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:59.073229    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:26:59.073684 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:26:59.073338    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:27:09.073784 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:09.073702    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:27:19.073584 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:19.073525    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:27:19.982988 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:19.982964    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:27:20.030995 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.030968    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmpmm\" (UniqueName: \"kubernetes.io/projected/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-kube-api-access-qmpmm\") pod \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") "
Sep 05 03:27:20.031141 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.031066    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-tls-certs\") pod \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\" (UID: \"d6e9be74-5b3c-4d4d-808d-145614d8ae1e\") "
Sep 05 03:27:20.033301 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.033269    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-kube-api-access-qmpmm" (OuterVolumeSpecName: "kube-api-access-qmpmm") pod "d6e9be74-5b3c-4d4d-808d-145614d8ae1e" (UID: "d6e9be74-5b3c-4d4d-808d-145614d8ae1e"). InnerVolumeSpecName "kube-api-access-qmpmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:27:20.033301 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.033286    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6e9be74-5b3c-4d4d-808d-145614d8ae1e" (UID: "d6e9be74-5b3c-4d4d-808d-145614d8ae1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:27:20.121392 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.121358    2553 generic.go:356] "Generic (PLEG): container finished" podID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerID="7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12" exitCode=0
Sep 05 03:27:20.121879 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.121405    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"
Sep 05 03:27:20.121879 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.121447    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" event={"ID":"d6e9be74-5b3c-4d4d-808d-145614d8ae1e","Type":"ContainerDied","Data":"7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12"}
Sep 05 03:27:20.121879 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.121518    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s" event={"ID":"d6e9be74-5b3c-4d4d-808d-145614d8ae1e","Type":"ContainerDied","Data":"e1846c45da09c61d2221b33598c9b542f658b8a2253a518c89724b52aa7d6a1d"}
Sep 05 03:27:20.121879 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.121554    2553 scope.go:117] "RemoveContainer" containerID="7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12"
Sep 05 03:27:20.131632 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.131614    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:27:20.131726 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.131633    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpmm\" (UniqueName: \"kubernetes.io/projected/d6e9be74-5b3c-4d4d-808d-145614d8ae1e-kube-api-access-qmpmm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:27:20.131863 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.131845    2553 scope.go:117] "RemoveContainer" containerID="7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12"
Sep 05 03:27:20.132135 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:27:20.132109    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12\": container with ID starting with 7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12 not found: ID does not exist" containerID="7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12"
Sep 05 03:27:20.132222 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.132143    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12"} err="failed to get container status \"7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12\": rpc error: code = NotFound desc = could not find container \"7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12\": container with ID starting with 7031c5d165b7c2bfa893d6b9ac996682d93dc73a3242e3f591721fd285b15e12 not found: ID does not exist"
Sep 05 03:27:20.143926 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.143902    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"]
Sep 05 03:27:20.146345 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.146324    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48c4g8s"]
Sep 05 03:27:20.393203 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:20.393114    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" path="/var/lib/kubelet/pods/d6e9be74-5b3c-4d4d-808d-145614d8ae1e/volumes"
Sep 05 03:27:51.977312 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.977273    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"]
Sep 05 03:27:51.977894 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.977772    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main"
Sep 05 03:27:51.977894 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.977794    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main"
Sep 05 03:27:51.977894 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.977882    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6e9be74-5b3c-4d4d-808d-145614d8ae1e" containerName="main"
Sep 05 03:27:51.980667 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.980645    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:51.983856 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.983830    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-ktffz\""
Sep 05 03:27:51.984104 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.984083    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 05 03:27:51.984194 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.984090    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 05 03:27:51.984194 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.984115    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-27csp\""
Sep 05 03:27:51.984194 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.984130    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 05 03:27:51.991479 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:51.991454    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"]
Sep 05 03:27:52.014912 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.014881    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.015056 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.014956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7226\" (UniqueName: \"kubernetes.io/projected/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-kube-api-access-k7226\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.116103 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.116065    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7226\" (UniqueName: \"kubernetes.io/projected/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-kube-api-access-k7226\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.116267 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.116146    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.118659 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.118632    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.123405 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.123374    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7226\" (UniqueName: \"kubernetes.io/projected/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-kube-api-access-k7226\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.292037 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.291996    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:52.562524 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:52.562499    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"]
Sep 05 03:27:52.568336 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:27:52.568308    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f0ed7c_b50c_4f3a_9cb1_3168467bcb10.slice/crio-db5673bfbf8bb03bff01dfd4ac96ea4c422929e31a1332f3b02c7390a64ed8c0 WatchSource:0}: Error finding container db5673bfbf8bb03bff01dfd4ac96ea4c422929e31a1332f3b02c7390a64ed8c0: Status 404 returned error can't find the container with id db5673bfbf8bb03bff01dfd4ac96ea4c422929e31a1332f3b02c7390a64ed8c0
Sep 05 03:27:53.279223 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:53.279183    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" event={"ID":"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10","Type":"ContainerStarted","Data":"d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33"}
Sep 05 03:27:53.279655 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:53.279274    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" event={"ID":"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10","Type":"ContainerStarted","Data":"db5673bfbf8bb03bff01dfd4ac96ea4c422929e31a1332f3b02c7390a64ed8c0"}
Sep 05 03:27:53.279655 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:53.279298    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:27:53.297616 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:27:53.297568    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" podStartSLOduration=2.297544175 podStartE2EDuration="2.297544175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:27:53.295189343 +0000 UTC m=+4183.494552335" watchObservedRunningTime="2026-09-05 03:27:53.297544175 +0000 UTC m=+4183.496907167"
Sep 05 03:28:20.335425 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:20.335388    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"]
Sep 05 03:28:24.285812 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:24.285783    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:28:26.813996 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:26.813951    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:28:35.357804 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:35.357713    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main" containerID="cri-o://3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6" gracePeriod=45
Sep 05 03:28:36.814184 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:36.814142    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:28:46.814396 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:46.814353    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:28:46.814900 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:46.814476    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:28:56.814064 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:28:56.814029    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:29:05.607279 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.607250    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:29:05.615795 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.615772    2553 generic.go:356] "Generic (PLEG): container finished" podID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerID="3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6" exitCode=0
Sep 05 03:29:05.615907 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.615803    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" event={"ID":"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37","Type":"ContainerDied","Data":"3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6"}
Sep 05 03:29:05.615907 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.615840    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5" event={"ID":"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37","Type":"ContainerDied","Data":"c370a18a3d8f9fc692571913e8bdbb56b46256f34baeee438b2ad014b858501b"}
Sep 05 03:29:05.615907 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.615865    2553 scope.go:117] "RemoveContainer" containerID="3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6"
Sep 05 03:29:05.615907 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.615804    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"
Sep 05 03:29:05.623630 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.623607    2553 scope.go:117] "RemoveContainer" containerID="3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6"
Sep 05 03:29:05.623970 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:29:05.623944    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6\": container with ID starting with 3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6 not found: ID does not exist" containerID="3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6"
Sep 05 03:29:05.624048 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.623980    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6"} err="failed to get container status \"3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6\": rpc error: code = NotFound desc = could not find container \"3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6\": container with ID starting with 3609b8e24f98738854337ea7b59903a0797af7845bf16db1df2ef448024b30f6 not found: ID does not exist"
Sep 05 03:29:05.732876 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.732853    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-tls-certs\") pod \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") "
Sep 05 03:29:05.733005 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.732913    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btwrb\" (UniqueName: \"kubernetes.io/projected/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-kube-api-access-btwrb\") pod \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\" (UID: \"d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37\") "
Sep 05 03:29:05.735007 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.734983    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" (UID: "d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:29:05.735120 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.735053    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-kube-api-access-btwrb" (OuterVolumeSpecName: "kube-api-access-btwrb") pod "d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" (UID: "d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37"). InnerVolumeSpecName "kube-api-access-btwrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:29:05.833754 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.833724    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:29:05.833754 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.833749    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwrb\" (UniqueName: \"kubernetes.io/projected/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37-kube-api-access-btwrb\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:29:05.954099 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.954065    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"]
Sep 05 03:29:05.959733 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:05.959709    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-kxwz5"]
Sep 05 03:29:06.394529 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:06.394492    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" path="/var/lib/kubelet/pods/d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37/volumes"
Sep 05 03:29:36.419104 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.419072    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"]
Sep 05 03:29:36.419737 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.419658    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main"
Sep 05 03:29:36.419737 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.419683    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main"
Sep 05 03:29:36.419871 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.419771    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ce8ee4-a3b6-4bfc-bd3b-c23a0e978f37" containerName="main"
Sep 05 03:29:36.422818 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.422796    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.425349 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.425330    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 05 03:29:36.426233 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.426207    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 05 03:29:36.426374 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.426236    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-j8dmg\""
Sep 05 03:29:36.426512 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.426489    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-8g5ll\""
Sep 05 03:29:36.426631 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.426497    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 05 03:29:36.431735 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.431713    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"]
Sep 05 03:29:36.493220 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.493190    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.493358 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.493313    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6mc\" (UniqueName: \"kubernetes.io/projected/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-kube-api-access-rl6mc\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.594116 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.594083    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6mc\" (UniqueName: \"kubernetes.io/projected/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-kube-api-access-rl6mc\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.594278 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.594123    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.596757 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.596729    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.605045 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.605023    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6mc\" (UniqueName: \"kubernetes.io/projected/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-kube-api-access-rl6mc\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:36.733135 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:36.733072    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:37.007613 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:37.007583    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"]
Sep 05 03:29:37.010503 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:29:37.010470    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc5492c_f3f9_4828_aa54_e5a7a475d4fa.slice/crio-08aebc2b7c90328c739ffeb33ec346991ec79f31d094c8c293411c122ffdcb4a WatchSource:0}: Error finding container 08aebc2b7c90328c739ffeb33ec346991ec79f31d094c8c293411c122ffdcb4a: Status 404 returned error can't find the container with id 08aebc2b7c90328c739ffeb33ec346991ec79f31d094c8c293411c122ffdcb4a
Sep 05 03:29:37.766332 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:37.766288    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" event={"ID":"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa","Type":"ContainerStarted","Data":"8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd"}
Sep 05 03:29:37.766725 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:37.766344    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" event={"ID":"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa","Type":"ContainerStarted","Data":"08aebc2b7c90328c739ffeb33ec346991ec79f31d094c8c293411c122ffdcb4a"}
Sep 05 03:29:37.766725 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:37.766377    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:29:37.783392 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:29:37.783347    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podStartSLOduration=1.783333443 podStartE2EDuration="1.783333443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:29:37.782046463 +0000 UTC m=+4287.981409457" watchObservedRunningTime="2026-09-05 03:29:37.783333443 +0000 UTC m=+4287.982696435"
Sep 05 03:30:04.647082 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:04.646993    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"]
Sep 05 03:30:08.772795 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:08.772756    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:30:14.286111 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:14.286073    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:30:19.669025 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:19.668974    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main" containerID="cri-o://d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33" gracePeriod=45
Sep 05 03:30:24.286026 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:24.285984    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:30:34.285878 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:34.285835    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:30:34.286268 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:34.285939    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:30:44.285830 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:44.285790    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:30:49.915762 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:49.915736    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:30:49.966457 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:49.966385    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7226\" (UniqueName: \"kubernetes.io/projected/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-kube-api-access-k7226\") pod \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") "
Sep 05 03:30:49.966624 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:49.966540    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-tls-certs\") pod \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\" (UID: \"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10\") "
Sep 05 03:30:49.968668 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:49.968638    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-kube-api-access-k7226" (OuterVolumeSpecName: "kube-api-access-k7226") pod "53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" (UID: "53f0ed7c-b50c-4f3a-9cb1-3168467bcb10"). InnerVolumeSpecName "kube-api-access-k7226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:30:49.968771 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:49.968687    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" (UID: "53f0ed7c-b50c-4f3a-9cb1-3168467bcb10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:30:50.067313 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.067274    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:30:50.067313 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.067307    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7226\" (UniqueName: \"kubernetes.io/projected/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10-kube-api-access-k7226\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:30:50.124857 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.124820    2553 generic.go:356] "Generic (PLEG): container finished" podID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerID="d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33" exitCode=0
Sep 05 03:30:50.125006 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.124868    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"
Sep 05 03:30:50.125006 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.124906    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" event={"ID":"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10","Type":"ContainerDied","Data":"d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33"}
Sep 05 03:30:50.125006 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.124962    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj" event={"ID":"53f0ed7c-b50c-4f3a-9cb1-3168467bcb10","Type":"ContainerDied","Data":"db5673bfbf8bb03bff01dfd4ac96ea4c422929e31a1332f3b02c7390a64ed8c0"}
Sep 05 03:30:50.125006 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.124986    2553 scope.go:117] "RemoveContainer" containerID="d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33"
Sep 05 03:30:50.134999 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.134978    2553 scope.go:117] "RemoveContainer" containerID="d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33"
Sep 05 03:30:50.135276 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:30:50.135255    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33\": container with ID starting with d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33 not found: ID does not exist" containerID="d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33"
Sep 05 03:30:50.135346 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.135282    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33"} err="failed to get container status \"d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33\": rpc error: code = NotFound desc = could not find container \"d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33\": container with ID starting with d251fbc336b066cedc4bd4dbfe777b24ac82f2474d68328ec8e5e687f0244b33 not found: ID does not exist"
Sep 05 03:30:50.149811 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.149001    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"]
Sep 05 03:30:50.153437 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.153413    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45m29fj"]
Sep 05 03:30:50.392763 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:30:50.392730    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" path="/var/lib/kubelet/pods/53f0ed7c-b50c-4f3a-9cb1-3168467bcb10/volumes"
Sep 05 03:31:01.502021 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.501981    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"]
Sep 05 03:31:01.502615 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.502334    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main"
Sep 05 03:31:01.502615 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.502345    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main"
Sep 05 03:31:01.502615 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.502419    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f0ed7c-b50c-4f3a-9cb1-3168467bcb10" containerName="main"
Sep 05 03:31:01.505403 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.505379    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.508053 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.508033    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 05 03:31:01.509038 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.509012    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 05 03:31:01.509157 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.509021    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-j5gxv\""
Sep 05 03:31:01.509157 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.509087    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 05 03:31:01.515020 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.514994    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"]
Sep 05 03:31:01.571456 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.571432    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.571594 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.571484    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.571594 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.571529    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.571691 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.571623    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.571691 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.571657    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwjpj\" (UniqueName: \"kubernetes.io/projected/b768090d-fddd-4f28-b22e-75152b26fadc-kube-api-access-wwjpj\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.571691 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.571678    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b768090d-fddd-4f28-b22e-75152b26fadc-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672251 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672218    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672372 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672277    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672372 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672310    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672372 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672341    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672546 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672377    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwjpj\" (UniqueName: \"kubernetes.io/projected/b768090d-fddd-4f28-b22e-75152b26fadc-kube-api-access-wwjpj\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672546 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672407    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b768090d-fddd-4f28-b22e-75152b26fadc-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672814 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672778    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672920 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672788    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.672920 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.672829    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.674631 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.674612    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.674860 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.674835    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b768090d-fddd-4f28-b22e-75152b26fadc-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.680036 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.680016    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwjpj\" (UniqueName: \"kubernetes.io/projected/b768090d-fddd-4f28-b22e-75152b26fadc-kube-api-access-wwjpj\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mcmwf\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:01.816713 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:01.816685    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:02.090351 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:02.087083    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"]
Sep 05 03:31:02.187587 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:02.187536    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" event={"ID":"b768090d-fddd-4f28-b22e-75152b26fadc","Type":"ContainerStarted","Data":"f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"}
Sep 05 03:31:02.187932 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:02.187615    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" event={"ID":"b768090d-fddd-4f28-b22e-75152b26fadc","Type":"ContainerStarted","Data":"e776034ac87624d73909ce3fcbe7e0dd72e3cb85e055f61941ffffacc45fa065"}
Sep 05 03:31:02.205390 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:02.205341    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" podStartSLOduration=1.205326713 podStartE2EDuration="1.205326713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:31:02.204410752 +0000 UTC m=+4372.403773756" watchObservedRunningTime="2026-09-05 03:31:02.205326713 +0000 UTC m=+4372.404689705"
Sep 05 03:31:02.817187 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:02.817148    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:11.817631 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:11.817589    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:11.829915 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:11.829880    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:12.245170 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:12.245089    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:46.194192 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.194108    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"]
Sep 05 03:31:46.203800 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:31:46.203754    2553 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:31:46.203800 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:31:46.203800 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:31:46.203800 ip-10-0-143-8 kubenswrapper[2553]:  > containerID="f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13" cmd=["/bin/sleep","15"]
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.203820    2553 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" message="" err=<
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]:  >
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:31:46.203833    2553 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]:         , stdout: , stderr: , exit code -1
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" containerName="main" containerID="cri-o://f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"
Sep 05 03:31:46.204053 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.203883    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" containerName="main" containerID="cri-o://f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13" gracePeriod=60
Sep 05 03:31:46.234955 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.234921    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 05 03:31:46.550241 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.550214    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:46.677019 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.676977    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-home\") pod \"b768090d-fddd-4f28-b22e-75152b26fadc\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") "
Sep 05 03:31:46.677202 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677042    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-tmp-dir\") pod \"b768090d-fddd-4f28-b22e-75152b26fadc\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") "
Sep 05 03:31:46.677202 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677079    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-dshm\") pod \"b768090d-fddd-4f28-b22e-75152b26fadc\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") "
Sep 05 03:31:46.677202 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677102    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwjpj\" (UniqueName: \"kubernetes.io/projected/b768090d-fddd-4f28-b22e-75152b26fadc-kube-api-access-wwjpj\") pod \"b768090d-fddd-4f28-b22e-75152b26fadc\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") "
Sep 05 03:31:46.677202 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677177    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b768090d-fddd-4f28-b22e-75152b26fadc-tls-certs\") pod \"b768090d-fddd-4f28-b22e-75152b26fadc\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") "
Sep 05 03:31:46.677202 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677196    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-model-cache\") pod \"b768090d-fddd-4f28-b22e-75152b26fadc\" (UID: \"b768090d-fddd-4f28-b22e-75152b26fadc\") "
Sep 05 03:31:46.677485 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677299    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-home" (OuterVolumeSpecName: "home") pod "b768090d-fddd-4f28-b22e-75152b26fadc" (UID: "b768090d-fddd-4f28-b22e-75152b26fadc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:46.677485 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677332    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b768090d-fddd-4f28-b22e-75152b26fadc" (UID: "b768090d-fddd-4f28-b22e-75152b26fadc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:46.677485 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677454    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-home\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:31:46.677485 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677470    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-tmp-dir\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:31:46.677686 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.677539    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b768090d-fddd-4f28-b22e-75152b26fadc" (UID: "b768090d-fddd-4f28-b22e-75152b26fadc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:46.679400 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.679367    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-dshm" (OuterVolumeSpecName: "dshm") pod "b768090d-fddd-4f28-b22e-75152b26fadc" (UID: "b768090d-fddd-4f28-b22e-75152b26fadc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:46.679515 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.679417    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b768090d-fddd-4f28-b22e-75152b26fadc-kube-api-access-wwjpj" (OuterVolumeSpecName: "kube-api-access-wwjpj") pod "b768090d-fddd-4f28-b22e-75152b26fadc" (UID: "b768090d-fddd-4f28-b22e-75152b26fadc"). InnerVolumeSpecName "kube-api-access-wwjpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:31:46.679515 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.679448    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b768090d-fddd-4f28-b22e-75152b26fadc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b768090d-fddd-4f28-b22e-75152b26fadc" (UID: "b768090d-fddd-4f28-b22e-75152b26fadc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:31:46.778940 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.778890    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-dshm\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:31:46.778940 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.778921    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwjpj\" (UniqueName: \"kubernetes.io/projected/b768090d-fddd-4f28-b22e-75152b26fadc-kube-api-access-wwjpj\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:31:46.778940 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.778933    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b768090d-fddd-4f28-b22e-75152b26fadc-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:31:46.778940 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:46.778942    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b768090d-fddd-4f28-b22e-75152b26fadc-model-cache\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:31:47.411283 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.411243    2553 generic.go:356] "Generic (PLEG): container finished" podID="b768090d-fddd-4f28-b22e-75152b26fadc" containerID="f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13" exitCode=0
Sep 05 03:31:47.411798 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.411292    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"
Sep 05 03:31:47.411798 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.411329    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" event={"ID":"b768090d-fddd-4f28-b22e-75152b26fadc","Type":"ContainerDied","Data":"f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"}
Sep 05 03:31:47.411798 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.411382    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf" event={"ID":"b768090d-fddd-4f28-b22e-75152b26fadc","Type":"ContainerDied","Data":"e776034ac87624d73909ce3fcbe7e0dd72e3cb85e055f61941ffffacc45fa065"}
Sep 05 03:31:47.411798 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.411406    2553 scope.go:117] "RemoveContainer" containerID="f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"
Sep 05 03:31:47.421451 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.421431    2553 scope.go:117] "RemoveContainer" containerID="f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"
Sep 05 03:31:47.421744 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:31:47.421726    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13\": container with ID starting with f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13 not found: ID does not exist" containerID="f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"
Sep 05 03:31:47.421812 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.421750    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13"} err="failed to get container status \"f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13\": rpc error: code = NotFound desc = could not find container \"f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13\": container with ID starting with f50627d3f3e6c0118349342feed8939f46ec8d576f7fa4822733bd56441f4e13 not found: ID does not exist"
Sep 05 03:31:47.433330 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.433302    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"]
Sep 05 03:31:47.436963 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:47.436938    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mcmwf"]
Sep 05 03:31:48.392892 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:48.392854    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" path="/var/lib/kubelet/pods/b768090d-fddd-4f28-b22e-75152b26fadc/volumes"
Sep 05 03:31:58.774232 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:58.774187    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:31:58.808980 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:31:58.808946    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"]
Sep 05 03:32:08.773293 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:08.773252    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:13.829833 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:13.829775    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main" containerID="cri-o://8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd" gracePeriod=45
Sep 05 03:32:18.773094 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:18.773048    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:18.773534 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:18.773173    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:32:28.772954 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:28.772909    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:38.773049 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:38.773006    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 03:32:44.082388 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.082352    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:32:44.163643 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.163525    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl6mc\" (UniqueName: \"kubernetes.io/projected/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-kube-api-access-rl6mc\") pod \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") "
Sep 05 03:32:44.163643 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.163644    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-tls-certs\") pod \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\" (UID: \"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa\") "
Sep 05 03:32:44.165968 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.165940    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-kube-api-access-rl6mc" (OuterVolumeSpecName: "kube-api-access-rl6mc") pod "4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" (UID: "4cc5492c-f3f9-4828-aa54-e5a7a475d4fa"). InnerVolumeSpecName "kube-api-access-rl6mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:32:44.166039 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.165966    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" (UID: "4cc5492c-f3f9-4828-aa54-e5a7a475d4fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:32:44.265258 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.265219    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6mc\" (UniqueName: \"kubernetes.io/projected/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-kube-api-access-rl6mc\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:32:44.265258 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.265258    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa-tls-certs\") on node \"ip-10-0-143-8.ec2.internal\" DevicePath \"\""
Sep 05 03:32:44.682029 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.681989    2553 generic.go:356] "Generic (PLEG): container finished" podID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerID="8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd" exitCode=0
Sep 05 03:32:44.682216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.682036    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"
Sep 05 03:32:44.682216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.682070    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" event={"ID":"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa","Type":"ContainerDied","Data":"8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd"}
Sep 05 03:32:44.682216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.682128    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj" event={"ID":"4cc5492c-f3f9-4828-aa54-e5a7a475d4fa","Type":"ContainerDied","Data":"08aebc2b7c90328c739ffeb33ec346991ec79f31d094c8c293411c122ffdcb4a"}
Sep 05 03:32:44.682216 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.682150    2553 scope.go:117] "RemoveContainer" containerID="8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd"
Sep 05 03:32:44.690709 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.690691    2553 scope.go:117] "RemoveContainer" containerID="8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd"
Sep 05 03:32:44.691000 ip-10-0-143-8 kubenswrapper[2553]: E0905 03:32:44.690976    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd\": container with ID starting with 8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd not found: ID does not exist" containerID="8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd"
Sep 05 03:32:44.691050 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.691006    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd"} err="failed to get container status \"8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd\": rpc error: code = NotFound desc = could not find container \"8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd\": container with ID starting with 8e8ccca3794e56099c944c7cffc7eb1421b4f5be33bea4497b56ac052bb16edd not found: ID does not exist"
Sep 05 03:32:44.699435 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.699410    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"]
Sep 05 03:32:44.703346 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:44.703323    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-bprrj"]
Sep 05 03:32:46.393895 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:32:46.393858    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" path="/var/lib/kubelet/pods/4cc5492c-f3f9-4828-aa54-e5a7a475d4fa/volumes"
Sep 05 03:33:28.055076 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:28.055034    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7978cdc4b-rn5tr_ae4b62c8-6434-4a5e-a9e2-b54c10be90b9/router/0.log"
Sep 05 03:33:29.283026 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:29.282994    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7978cdc4b-rn5tr_ae4b62c8-6434-4a5e-a9e2-b54c10be90b9/router/0.log"
Sep 05 03:33:30.296587 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:30.296528    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-c26gd_4caf73f9-59bd-4e2f-ace3-39b25bc5e352/authorino/0.log"
Sep 05 03:33:30.312391 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:30.312362    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-4d9fp_5084691f-d6e5-4464-b80c-45e447756f69/manager/0.log"
Sep 05 03:33:30.327779 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:30.327751    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-msqjr_ce9fdeba-e8a3-4b60-92de-a779b11f760e/manager/1.log"
Sep 05 03:33:30.327890 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:30.327762    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-msqjr_ce9fdeba-e8a3-4b60-92de-a779b11f760e/manager/0.log"
Sep 05 03:33:39.309065 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:39.309026    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2fzp6_a857aaf5-62f1-4114-a4e3-2b1ecad4ef84/global-pull-secret-syncer/0.log"
Sep 05 03:33:39.514295 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:39.514261    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wxqrh_c4059ab2-be92-4752-8b64-e8b5044bfbff/konnectivity-agent/0.log"
Sep 05 03:33:39.586222 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:39.586155    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-8.ec2.internal_e56369bb26fe910c3debbb11d5297fda/haproxy/0.log"
Sep 05 03:33:44.384981 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:44.384948    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-c26gd_4caf73f9-59bd-4e2f-ace3-39b25bc5e352/authorino/0.log"
Sep 05 03:33:44.416228 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:44.416196    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-4d9fp_5084691f-d6e5-4464-b80c-45e447756f69/manager/0.log"
Sep 05 03:33:44.450487 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:44.450462    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-msqjr_ce9fdeba-e8a3-4b60-92de-a779b11f760e/manager/1.log"
Sep 05 03:33:44.451421 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:44.451399    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-msqjr_ce9fdeba-e8a3-4b60-92de-a779b11f760e/manager/0.log"
Sep 05 03:33:46.352872 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.352845    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/alertmanager/0.log"
Sep 05 03:33:46.392897 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.392872    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/config-reloader/0.log"
Sep 05 03:33:46.436179 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.436110    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/kube-rbac-proxy-web/0.log"
Sep 05 03:33:46.477349 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.477322    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/kube-rbac-proxy/0.log"
Sep 05 03:33:46.518432 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.518411    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/kube-rbac-proxy-metric/0.log"
Sep 05 03:33:46.555031 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.555008    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/prom-label-proxy/0.log"
Sep 05 03:33:46.591254 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.591236    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_731a132e-6ee1-4616-ac94-52dd63655ef4/init-config-reloader/0.log"
Sep 05 03:33:46.675367 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.675339    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gv6c5_3266367a-da1d-42be-8f8f-b7d05cf1a9c4/kube-state-metrics/0.log"
Sep 05 03:33:46.698414 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.698354    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gv6c5_3266367a-da1d-42be-8f8f-b7d05cf1a9c4/kube-rbac-proxy-main/0.log"
Sep 05 03:33:46.719214 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.719188    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gv6c5_3266367a-da1d-42be-8f8f-b7d05cf1a9c4/kube-rbac-proxy-self/0.log"
Sep 05 03:33:46.746414 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.746395    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd5f8c8dd-4zkgm_1142348b-19ce-4c4c-b173-176ca6ee6130/metrics-server/0.log"
Sep 05 03:33:46.778122 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.778099    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-xcm7r_27cecc77-d095-4df3-bc52-29b54fd193d0/monitoring-plugin/0.log"
Sep 05 03:33:46.810067 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.810036    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tfbs_42e4d112-aa98-4f8e-931a-487fcc4fa5c9/node-exporter/0.log"
Sep 05 03:33:46.834520 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.834501    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tfbs_42e4d112-aa98-4f8e-931a-487fcc4fa5c9/kube-rbac-proxy/0.log"
Sep 05 03:33:46.856192 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:46.856171    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tfbs_42e4d112-aa98-4f8e-931a-487fcc4fa5c9/init-textfile/0.log"
Sep 05 03:33:47.092643 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.092604    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9d6fc_82310afe-e173-4719-a2df-0333fb883f8e/kube-rbac-proxy-main/0.log"
Sep 05 03:33:47.114731 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.114706    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9d6fc_82310afe-e173-4719-a2df-0333fb883f8e/kube-rbac-proxy-self/0.log"
Sep 05 03:33:47.138400 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.138373    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9d6fc_82310afe-e173-4719-a2df-0333fb883f8e/openshift-state-metrics/0.log"
Sep 05 03:33:47.191236 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.191207    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/prometheus/0.log"
Sep 05 03:33:47.212375 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.212354    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/config-reloader/0.log"
Sep 05 03:33:47.240996 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.240935    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/thanos-sidecar/0.log"
Sep 05 03:33:47.286576 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.286543    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/kube-rbac-proxy-web/0.log"
Sep 05 03:33:47.319400 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.319377    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/kube-rbac-proxy/0.log"
Sep 05 03:33:47.345785 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.345762    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/kube-rbac-proxy-thanos/0.log"
Sep 05 03:33:47.373707 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.373686    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4634099a-9a65-41d2-992e-f6651990d75a/init-config-reloader/0.log"
Sep 05 03:33:47.423285 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.423256    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-mq44t_a5977287-5f4c-4ea2-b4e8-424f7f329c4c/prometheus-operator/0.log"
Sep 05 03:33:47.446346 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.446319    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-mq44t_a5977287-5f4c-4ea2-b4e8-424f7f329c4c/kube-rbac-proxy/0.log"
Sep 05 03:33:47.476139 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.476117    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-zk6kc_ea0af986-d6ca-4e86-b651-180096161852/prometheus-operator-admission-webhook/0.log"
Sep 05 03:33:47.601706 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.601667    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/thanos-query/0.log"
Sep 05 03:33:47.627613 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.627556    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy-web/0.log"
Sep 05 03:33:47.657353 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.657332    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy/0.log"
Sep 05 03:33:47.679729 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.679705    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/prom-label-proxy/0.log"
Sep 05 03:33:47.705957 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.705935    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy-rules/0.log"
Sep 05 03:33:47.733882 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:47.733859    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64bfc7bdcb-nqvfn_510f85ce-e160-4340-b969-857c4c9a15e4/kube-rbac-proxy-metrics/0.log"
Sep 05 03:33:48.974641 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:48.974614    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-qvr68_6dd08154-87bf-435e-b2cc-827a2419b85f/networking-console-plugin/0.log"
Sep 05 03:33:50.005670 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.005628    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"]
Sep 05 03:33:50.006162 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.006143    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main"
Sep 05 03:33:50.006234 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.006164    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main"
Sep 05 03:33:50.006234 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.006210    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" containerName="main"
Sep 05 03:33:50.006234 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.006220    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" containerName="main"
Sep 05 03:33:50.006384 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.006311    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc5492c-f3f9-4828-aa54-e5a7a475d4fa" containerName="main"
Sep 05 03:33:50.006384 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.006325    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b768090d-fddd-4f28-b22e-75152b26fadc" containerName="main"
Sep 05 03:33:50.009672 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.009648    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.012369 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.012346    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g75mm\"/\"kube-root-ca.crt\""
Sep 05 03:33:50.012499 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.012440    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g75mm\"/\"default-dockercfg-js4pb\""
Sep 05 03:33:50.013284 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.013266    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g75mm\"/\"openshift-service-ca.crt\""
Sep 05 03:33:50.020794 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.020774    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"]
Sep 05 03:33:50.030798 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.030777    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-rvl98_965dc0fc-5506-42dd-a079-0d548fdd31c4/download-server/0.log"
Sep 05 03:33:50.095709 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.095677    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-podres\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.095842 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.095717    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbb7d\" (UniqueName: \"kubernetes.io/projected/0fa12638-bf50-43e9-b220-a4ab7c96a48e-kube-api-access-gbb7d\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.095842 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.095762    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-proc\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.095842 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.095791    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-sys\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.095976 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.095842    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-lib-modules\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.196861 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.196830    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-lib-modules\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.196980 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.196895    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-podres\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.196980 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.196916    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbb7d\" (UniqueName: \"kubernetes.io/projected/0fa12638-bf50-43e9-b220-a4ab7c96a48e-kube-api-access-gbb7d\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.196980 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.196939    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-proc\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.196980 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.196957    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-sys\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.197132 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.197021    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-sys\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.197132 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.197017    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-lib-modules\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.197132 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.197044    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-proc\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.197132 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.197079    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fa12638-bf50-43e9-b220-a4ab7c96a48e-podres\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.204308 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.204281    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbb7d\" (UniqueName: \"kubernetes.io/projected/0fa12638-bf50-43e9-b220-a4ab7c96a48e-kube-api-access-gbb7d\") pod \"perf-node-gather-daemonset-tzq5x\" (UID: \"0fa12638-bf50-43e9-b220-a4ab7c96a48e\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.320606 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.320586    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:50.590724 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.590697    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"]
Sep 05 03:33:50.592636 ip-10-0-143-8 kubenswrapper[2553]: W0905 03:33:50.592606    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0fa12638_bf50_43e9_b220_a4ab7c96a48e.slice/crio-d0c6863705181ebe5a7f3a6ee2ed3ae0f18720a83e836de0b9c01a09046cf8d7 WatchSource:0}: Error finding container d0c6863705181ebe5a7f3a6ee2ed3ae0f18720a83e836de0b9c01a09046cf8d7: Status 404 returned error can't find the container with id d0c6863705181ebe5a7f3a6ee2ed3ae0f18720a83e836de0b9c01a09046cf8d7
Sep 05 03:33:50.988664 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.988570    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x" event={"ID":"0fa12638-bf50-43e9-b220-a4ab7c96a48e","Type":"ContainerStarted","Data":"489da13c4e25136c74a59ee6b6221c9a97293c6b54ab6e75e9d1015979407391"}
Sep 05 03:33:50.988664 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.988635    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x" event={"ID":"0fa12638-bf50-43e9-b220-a4ab7c96a48e","Type":"ContainerStarted","Data":"d0c6863705181ebe5a7f3a6ee2ed3ae0f18720a83e836de0b9c01a09046cf8d7"}
Sep 05 03:33:50.988864 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:50.988667    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:51.008089 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:51.008040    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x" podStartSLOduration=2.008025306 podStartE2EDuration="2.008025306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:33:51.007332223 +0000 UTC m=+4541.206695216" watchObservedRunningTime="2026-09-05 03:33:51.008025306 +0000 UTC m=+4541.207388660"
Sep 05 03:33:51.734005 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:51.733962    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcsnn_b72ef229-be3c-4e6e-8d1f-59488d8a46ff/dns/0.log"
Sep 05 03:33:51.756042 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:51.756017    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcsnn_b72ef229-be3c-4e6e-8d1f-59488d8a46ff/kube-rbac-proxy/0.log"
Sep 05 03:33:51.856715 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:51.856677    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-96gdl_0d380942-90b8-482d-a917-d4d3435fb445/dns-node-resolver/0.log"
Sep 05 03:33:52.481267 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:52.481228    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rsqww_a2a60161-98fa-4b9d-87a1-457e09a1b7fa/node-ca/0.log"
Sep 05 03:33:53.938500 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:53.938467    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7978cdc4b-rn5tr_ae4b62c8-6434-4a5e-a9e2-b54c10be90b9/router/0.log"
Sep 05 03:33:54.473964 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:54.473933    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gp8cp_9b4cb734-f2b5-4a9e-b322-d91f90d4e05a/serve-healthcheck-canary/0.log"
Sep 05 03:33:54.988328 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:54.988275    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jblb8_d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5/kube-rbac-proxy/0.log"
Sep 05 03:33:55.009494 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:55.009463    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jblb8_d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5/exporter/0.log"
Sep 05 03:33:55.030771 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:55.030748    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jblb8_d86d4179-8e90-4785-8b3d-0c4ecb9cf1b5/extractor/0.log"
Sep 05 03:33:57.003059 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:57.003030    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-tzq5x"
Sep 05 03:33:58.056320 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:58.056291    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54b595bc86-765tw_ee9b3946-f9a1-48b3-bab6-dfb23af859c8/manager/0.log"
Sep 05 03:33:58.106401 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:33:58.106374    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wqc59_bc9c28ec-6277-4a55-95ac-6b80cd9208a8/openshift-lws-operator/0.log"
Sep 05 03:34:01.867037 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:01.867007    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7978cdc4b-rn5tr_ae4b62c8-6434-4a5e-a9e2-b54c10be90b9/router/0.log"
Sep 05 03:34:03.452138 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:03.452094    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7978cdc4b-rn5tr_ae4b62c8-6434-4a5e-a9e2-b54c10be90b9/router/0.log"
Sep 05 03:34:07.916162 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:07.916126    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:34:07.938039 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:07.938006    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/egress-router-binary-copy/0.log"
Sep 05 03:34:07.961054 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:07.961024    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/cni-plugins/0.log"
Sep 05 03:34:07.983787 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:07.983759    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/bond-cni-plugin/0.log"
Sep 05 03:34:08.004341 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:08.004309    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/routeoverride-cni/0.log"
Sep 05 03:34:08.026298 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:08.026261    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/whereabouts-cni-bincopy/0.log"
Sep 05 03:34:08.048354 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:08.048326    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zmrls_d6862640-b4c3-47c5-a10b-13d7df1a9948/whereabouts-cni/0.log"
Sep 05 03:34:08.095247 ip-10-0-143-8 kubenswrapper[2553]: I0905 03:34:08.095217    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f2sf6_0437b2bf-4237-4912-bc21-8dd6e444cfda/kube-multus/0.log"