Sep 17 23:03:57.661454 ip-10-0-143-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:03:57.661465 ip-10-0-143-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:03:57.661472 ip-10-0-143-250 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:03:57.661678 ip-10-0-143-250 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:04:07.760748 ip-10-0-143-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:04:07.760765 ip-10-0-143-250 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e0e73169104b4a7f91ce77c88607efa3 --
Sep 17 23:05:56.153033 ip-10-0-143-250 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:05:59.432023 ip-10-0-143-250 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:05:59.432023 ip-10-0-143-250 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:05:59.432023 ip-10-0-143-250 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:05:59.432023 ip-10-0-143-250 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:05:59.432023 ip-10-0-143-250 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:05:59.432847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.432101    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:05:59.435865 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435849    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:05:59.435865 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435865    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435869    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435873    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435876    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435879    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435881    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435884    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435887    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435889    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435892    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435895    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435901    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435904    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435907    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435909    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435913    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435917    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435919    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435922    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435931    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:05:59.435924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435934    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435937    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435939    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435942    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435944    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435947    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435949    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435952    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435954    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435957    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435959    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435962    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435964    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435966    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435969    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435971    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435974    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435976    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435979    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435981    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:05:59.436384 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435983    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435986    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435989    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435991    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435994    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435996    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.435999    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436001    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436004    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436008    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436010    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436013    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436016    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436018    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436021    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436024    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436027    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436029    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436032    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:05:59.436914 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436034    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436037    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436040    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436042    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436044    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436047    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436050    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436053    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436055    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436058    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436060    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436064    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436066    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436068    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436071    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436073    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436076    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436078    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436081    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436083    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:05:59.437363 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436086    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436089    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436092    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436094    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436096    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436099    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436102    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436105    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436108    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436110    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436113    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436115    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436118    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436121    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436123    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436125    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436128    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436132    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436135    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:05:59.437849 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436829    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436836    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436839    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436843    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436845    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436849    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436851    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436854    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436857    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436859    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436862    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436865    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436867    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436871    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436876    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436879    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436882    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436885    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436888    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436891    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:05:59.438301 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436893    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436896    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436899    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436902    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436905    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436908    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436911    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436915    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436918    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436921    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436924    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436927    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436929    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436931    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436934    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436936    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436939    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436941    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436948    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:05:59.438811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436951    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436953    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436956    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436959    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436962    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436965    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436967    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436970    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436973    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436976    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436978    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436981    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436983    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436986    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436988    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436991    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436993    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436996    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.436998    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437001    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:05:59.439256 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437003    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437005    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437008    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437010    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437013    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437015    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437018    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437021    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437023    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437025    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437028    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437030    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437033    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437036    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437038    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437040    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437043    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437046    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437048    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437052    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:05:59.439787 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437055    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437058    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437060    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437064    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437067    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437070    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437073    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437076    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437079    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437081    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437084    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437086    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437089    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437091    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437094    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437097    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437099    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437102    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437105    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:05:59.440269 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437107    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437455    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437460    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437462    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437465    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437468    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437471    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437473    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437476    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437481    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437484    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437486    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437489    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437492    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437495    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437497    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437501    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437505    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437508    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437511    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:05:59.440745 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437513    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437516    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437519    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437521    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437524    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437526    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437528    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437531    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437533    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437536    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437538    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437540    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437544    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437547    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437550    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437553    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437556    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437558    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437561    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:05:59.441232 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437564    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437567    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437570    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437573    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437576    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437580    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437583    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437586    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437589    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437593    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437595    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437598    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437601    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437603    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437606    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437609    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437611    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437614    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437618    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:05:59.441692 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437620    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437623    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437626    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437628    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437631    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437634    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437636    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437639    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437641    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437643    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437646    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437648    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437651    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437654    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437656    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437660    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437662    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437665    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437668    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437670    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:05:59.442174 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437673    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437675    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437677    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437680    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437682    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437685    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437687    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437690    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437692    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437695    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437697    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437700    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437702    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437705    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437707    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437710    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437712    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437730    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437733    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437736    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:05:59.442767 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437738    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.437741    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437813    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437850    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437857    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437862    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437867    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437871    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437875    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437879    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437884    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437887    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437891    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437894    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437897    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437899    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437902    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437905    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437908    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437911    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437916    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437919    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437922    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437925    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:05:59.443238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437928    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437931    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437934    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437937    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437940    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437943    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437946    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437949    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437951    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437954    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437958    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437961    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437964    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437967    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437970    2559 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437974    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437979    2559 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437982    2559 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437986    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437989    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437992    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437995    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.437998    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438002    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438004    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:05:59.443823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438007    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438010    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438013    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438016    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438019    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438022    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438024    2559 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438028    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438031    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438034    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438037    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438040    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438043    2559 flags.go:64] FLAG: --help="false"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438046    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438049    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438052    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438055    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438058    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438061    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438064    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438067    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438070    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438073    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:05:59.444400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438076    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438079    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438083    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438086    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438089    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438092    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438095    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438097    2559 flags.go:64] FLAG: --lock-file=""
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438100    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438103    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438106    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438111    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438114    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438117    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438119    2559 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438122    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438125    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438128    2559 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438131    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438135    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438138    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438142    2559 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438145    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438148    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438150    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:05:59.445021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438153    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438156    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438159    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438162    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438169    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438171    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438174    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438177    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438181    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438186    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438190    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438193    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438196    2559 flags.go:64] FLAG: --port="10250"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438199    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438202    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b56c5f802bcb758a"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438205    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438208    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438211    2559 flags.go:64] FLAG: --register-node="true"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438214    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438217    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438220    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438222    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438225    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438229    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438231    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:05:59.445643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438234    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438237    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438240    2559 flags.go:64] FLAG: --runonce="false"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438243    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438245    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438248    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438251    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438254    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438257    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438260    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438263    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438266    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438268    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438272    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438275    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438278    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438281    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438285    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438290    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438293    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438295    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438830    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438833    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438836    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438840    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:05:59.446302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438843    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438846    2559 flags.go:64] FLAG: --v="2"
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438850    2559 flags.go:64] FLAG: --version="false"
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438853    2559 flags.go:64] FLAG: --vmodule=""
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438857    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.438860    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438972    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438976    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438979    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438982    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438985    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438988    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438991    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438994    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.438997    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439000    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439003    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439005    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439007    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439010    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439014    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439017    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:05:59.446902 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439020    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439022    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439026    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439028    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439031    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439033    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439036    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439039    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439041    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439044    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439046    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439049    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439051    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439054    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439056    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439059    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439061    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439064    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439066    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439069    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:05:59.447422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439071    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439074    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439077    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439079    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439082    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439084    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439087    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439089    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439091    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439094    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439099    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439103    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439106    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439110    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439114    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439118    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439121    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439124    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:05:59.447905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439126    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439129    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439132    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439134    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439137    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439139    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439142    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439144    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439147    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439149    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439152    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439154    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439157    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439159    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439162    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439164    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439167    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439169    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439172    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439174    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:05:59.448339 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439177    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439179    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439182    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439184    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439188    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439191    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439193    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439195    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439198    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439200    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439207    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439209    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439212    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439214    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439216    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439220    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439224    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439227    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439229    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:05:59.448870 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439232    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439235    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439238    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439240    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439243    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.439246    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.439802    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.446668    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.446683    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.446696    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.446702    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446768    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446774    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446778    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446781    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:05:59.449313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446784    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446787    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446790    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446793    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446795    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446798    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446802    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446806    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446810    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446813    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446817    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446820    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446823    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446826    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446829    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446832    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446834    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446837    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446840    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:05:59.449705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446843    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446845    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446848    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446850    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446852    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446855    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446860    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446864    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446867    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446870    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446873    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446876    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446878    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446881    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446883    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446886    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446889    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446891    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446893    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:05:59.450300 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446896    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446899    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446901    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446905    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446907    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446910    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446914    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446918    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446921    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446923    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446926    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446928    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446931    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446933    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446935    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446938    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446940    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446943    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446945    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446948    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:05:59.450925 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446951    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446953    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446956    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446959    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446962    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446964    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446966    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446969    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446971    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446974    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446976    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446978    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446981    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446983    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446986    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446988    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446991    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446994    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446996    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:05:59.451409 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.446999    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447001    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447003    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447006    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447009    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447012    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447014    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447017    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447019    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447022    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447024    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447027    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447029    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447032    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447034    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447036    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447039    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:05:59.452155 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447041    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.447046    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447151    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447156    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447158    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447161    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447164    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447166    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447169    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447171    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447174    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447176    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447178    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447181    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447184    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:05:59.452970 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447186    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447189    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447191    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447194    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447197    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447199    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447202    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447204    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447207    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447209    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447211    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447214    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447216    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447220    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447223    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447226    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447229    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447231    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447234    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:05:59.453405 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447236    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447239    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447242    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447244    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447246    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447249    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447251    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447254    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447256    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447259    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447261    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447264    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447267    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447269    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447272    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447274    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447276    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447279    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447282    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447284    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:05:59.453905 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447287    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447289    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447291    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447294    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447296    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447299    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447301    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447305    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447308    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447311    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447313    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447315    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447318    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447320    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447323    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447325    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447327    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447330    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447332    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:05:59.454440 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447335    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447337    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447340    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447342    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447344    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447347    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447350    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447352    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447355    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447357    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447360    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447362    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447364    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447367    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447369    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447372    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447374    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447376    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447379    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447384    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:05:59.454969 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447386    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447389    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447391    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447394    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447397    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447400    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447404    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:05:59.447407    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.447412    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.447511    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.449611    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.450456    2559 server.go:1019] "Starting client certificate rotation"
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.450566    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:05:59.455491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.450618    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:05:59.455888 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.455862    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:05:59.457652 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.457629    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:05:59.466652 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.466637    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:05:59.468997 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.468980    2559 log.go:25] "Validated CRI v1 image API"
Sep 17 23:05:59.469795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.469779    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:05:59.472057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.472032    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bd68dceb-b5c0-4ae2-8a5b-d24e9f5fd3a8:/dev/nvme0n1p4 efd95a61-01e4-479b-b5bc-f8591f26b52a:/dev/nvme0n1p3]
Sep 17 23:05:59.472117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.472057    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:05:59.477179 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.477078    2559 manager.go:216] Machine: {Timestamp:2026-09-17 23:05:59.47661606 +0000 UTC m=+1.950559033 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3206674 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec256f78fc21b08b40f3ceaf569d412d SystemUUID:ec256f78-fc21-b08b-40f3-ceaf569d412d BootID:e0e73169-104b-4a7f-91ce-77c88607efa3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:b4:32:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:b4:32:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:d2:1b:a6:a2:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:05:59.477179 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.477174    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:05:59.477305 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.477273    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:05:59.478911 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.478889    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:05:59.479046 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.478913    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:05:59.480103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.480092    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:05:59.480135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.480106    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:05:59.480135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.480113    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:05:59.480135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.480123    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:05:59.480249 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.480190    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:05:59.483908 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.483892    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:05:59.484709 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484697    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:05:59.484831 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484822    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:05:59.484867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484848    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:05:59.484867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484858    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:05:59.484937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484872    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:05:59.484937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484881    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:05:59.484937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.484889    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:05:59.485296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.485283    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:05:59.485327 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.485309    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:05:59.487298 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.487270    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:05:59.492594 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.492575    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:05:59.492684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.492613    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:05:59.493847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493833    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493859    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493869    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493876    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493888    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493895    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493913    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493921    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:05:59.493927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493930    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:05:59.494176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493938    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:05:59.494176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493950    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:05:59.494176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493959    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:05:59.494176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493990    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:05:59.494176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.493999    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:05:59.495995 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.495970    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:05:59.496073 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.495977    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:05:59.496127 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.496104    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:05:59.497713 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.497698    2559 server.go:1268] "Started kubelet"
Sep 17 23:05:59.497810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.497764    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:05:59.497867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.497800    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:05:59.497867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.497862    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:05:59.498398 ip-10-0-143-250 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:05:59.499187 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.499138    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:05:59.499652 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.499635    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:05:59.503420 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.503403    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:05:59.505743 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.505711    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:05:59.505743 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.505731    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:05:59.505895 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.505852    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:05:59.506425 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506407    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:05:59.506425 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506428    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:05:59.506564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506407    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:05:59.506564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506538    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:05:59.506564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506548    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:05:59.506713 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506659    2559 factory.go:55] Registering systemd factory
Sep 17 23:05:59.506713 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506670    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:05:59.506880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506865    2559 factory.go:153] Registering CRI-O factory
Sep 17 23:05:59.506880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506879    2559 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:05:59.506965 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506928    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:05:59.506965 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506955    2559 factory.go:103] Registering Raw factory
Sep 17 23:05:59.507029 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.506973    2559 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:05:59.507465 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.507453    2559 manager.go:349] Starting recovery of all containers
Sep 17 23:05:59.508938 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.508913    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:05:59.509029 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.509013    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:05:59.517735 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.517710    2559 manager.go:354] Recovery completed
Sep 17 23:05:59.535562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.535515    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zdnkz"
Sep 17 23:05:59.537018 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.536994    2559 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 23:05:59.540399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.540385    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.542673 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.542657    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zdnkz"
Sep 17 23:05:59.543231 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543217    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.543310 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543243    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.543310 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543253    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.543769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543753    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.543861 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543757    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:05:59.543861 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543833    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:05:59.543861 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.543854    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:05:59.544541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.544523    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.544610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.544554    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.544610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.544566    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.546523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.546506    2559 policy_none.go:49] "None policy: Start"
Sep 17 23:05:59.546523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.546522    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:05:59.546640 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.546532    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:05:59.548662 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.548648    2559 policy_none.go:47] "Start"
Sep 17 23:05:59.588256 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588243    2559 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:05:59.588348 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.588284    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:05:59.588348 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588293    2559 server.go:87] "Starting device plugin registration server"
Sep 17 23:05:59.588537 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588525    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:05:59.588578 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588550    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:05:59.588703 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588686    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:05:59.588805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588793    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:05:59.588866 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.588809    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:05:59.589230 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.589202    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:05:59.589318 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.589254    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:05:59.631432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.631402    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:05:59.632558 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.632543    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:05:59.632621 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.632588    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:05:59.632621 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.632607    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:05:59.632707 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.632649    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:05:59.635035 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.635021    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:05:59.688699 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.688636    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.689854 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.689839    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.689919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.689866    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.689919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.689876    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.689919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.689900    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.699135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.699120    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.699208 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.699138    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-250.ec2.internal\": node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:05:59.726771 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.726751    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:05:59.733739 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.733709    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"]
Sep 17 23:05:59.733801 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.733778    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.734509 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.734493    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.734567 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.734522    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.734567 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.734532    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.736865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.736852    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.737007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.736995    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.737041 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737022    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.737511 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737491    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.737511 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737499    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.737646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737518    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.737646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737520    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.737646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737528    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.737646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.737534    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.739896 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.739880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.739974 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.739909    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:05:59.740487 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.740472    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:05:59.740554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.740504    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:05:59.740554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.740518    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:05:59.755207 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.755188    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-250.ec2.internal\" not found" node="ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.759493 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.759479    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-250.ec2.internal\" not found" node="ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.807530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.807509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a14bb4ae145cfdc960ab64cc87036398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal\" (UID: \"a14bb4ae145cfdc960ab64cc87036398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.807619 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.807536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55c41d7938cdbb09331b64496abe4d6c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-250.ec2.internal\" (UID: \"55c41d7938cdbb09331b64496abe4d6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.807619 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.807558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a14bb4ae145cfdc960ab64cc87036398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal\" (UID: \"a14bb4ae145cfdc960ab64cc87036398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.827021 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.827004    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:05:59.908095 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.908080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a14bb4ae145cfdc960ab64cc87036398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal\" (UID: \"a14bb4ae145cfdc960ab64cc87036398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.908160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.908103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a14bb4ae145cfdc960ab64cc87036398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal\" (UID: \"a14bb4ae145cfdc960ab64cc87036398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.908160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.908123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55c41d7938cdbb09331b64496abe4d6c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-250.ec2.internal\" (UID: \"55c41d7938cdbb09331b64496abe4d6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.908220 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.908167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55c41d7938cdbb09331b64496abe4d6c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-250.ec2.internal\" (UID: \"55c41d7938cdbb09331b64496abe4d6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.908220 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.908202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a14bb4ae145cfdc960ab64cc87036398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal\" (UID: \"a14bb4ae145cfdc960ab64cc87036398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.908279 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:05:59.908180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a14bb4ae145cfdc960ab64cc87036398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal\" (UID: \"a14bb4ae145cfdc960ab64cc87036398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:05:59.927213 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:05:59.927185    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:06:00.027878 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.027860    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:06:00.059013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.058998    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:06:00.063294 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.063280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"
Sep 17 23:06:00.128933 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.128912    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:06:00.229396 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.229377    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-250.ec2.internal\" not found"
Sep 17 23:06:00.306283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.306242    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:00.406512 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.406492    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal"
Sep 17 23:06:00.418995 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.418979    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:06:00.420653 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.420642    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal"
Sep 17 23:06:00.428926 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.428912    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:06:00.450831 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.450808    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:06:00.451239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.450930    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:06:00.485916 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.485900    2559 apiserver.go:52] "Watching apiserver"
Sep 17 23:06:00.493380 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.493362    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:06:00.493694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.493674    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-n87zb","openshift-dns/node-resolver-c6w6w","openshift-image-registry/node-ca-d98wj","openshift-multus/multus-sxdqn","openshift-network-diagnostics/network-check-target-h7pb7","kube-system/global-pull-secret-syncer-4qpfx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal","openshift-multus/multus-additional-cni-plugins-m2bsm","openshift-multus/network-metrics-daemon-kd746","openshift-network-operator/iptables-alerter-5bsf7","openshift-ovn-kubernetes/ovnkube-node-nq45x","kube-system/konnectivity-agent-sb77f","kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"]
Sep 17 23:06:00.496199 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.496183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.499239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bht7f\""
Sep 17 23:06:00.499239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499190    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.499239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499189    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:06:00.499239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499210    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:06:00.499658 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499272    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:06:00.499658 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499208    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.499805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.499790    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.501738 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.501708    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.501738 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.501735    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.501848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.501763    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-czpdx\""
Sep 17 23:06:00.501996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.501983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.503746 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.503728    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:06:00.503826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.503818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.503883 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.503863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.504051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.504035    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c2mkq\""
Sep 17 23:06:00.504283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.504270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.506075 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.506059    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:06:00.506477 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.506463    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:06:00.506563 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.506549    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:00.506628 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.506615    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:06:00.506666 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.506606    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:00.507298 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.507281    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.507369 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.507333    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:06:00.507419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.507365    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.507419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.507374    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqvrf\""
Sep 17 23:06:00.507498 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.507431    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:06:00.508811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.508797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.510603 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.510690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-log-socket\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.510690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:06:00.510793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.510793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510695    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.510793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-system-cni-dir\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.510793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af2fe43e-8f00-4eeb-b27d-1ff01d211599-serviceca\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.510939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510807    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.510939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-systemd-units\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.510939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-run-netns\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.510939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-etc-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.510939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvjc\" (UniqueName: \"kubernetes.io/projected/647f02eb-3c4a-4b27-8d3a-9892a08754df-kube-api-access-2dvjc\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.510939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-cni-bin\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-cni-netd\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.510985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1300ca1-628e-43b9-8dfc-fcd0d197a975-host-slash\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmg8\" (UniqueName: \"kubernetes.io/projected/e1300ca1-628e-43b9-8dfc-fcd0d197a975-kube-api-access-zlmg8\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511022    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ltsbr\""
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af2fe43e-8f00-4eeb-b27d-1ff01d211599-host\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-slash\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-node-log\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511132    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovn-node-metrics-cert\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16511250-0c7e-4b12-841b-afeba2af4bc8-hosts-file\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgqs\" (UniqueName: \"kubernetes.io/projected/af2fe43e-8f00-4eeb-b27d-1ff01d211599-kube-api-access-dhgqs\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.511192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-systemd\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-env-overrides\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8tb\" (UniqueName: \"kubernetes.io/projected/7e63ef48-0367-4e32-b7af-f2f23c7dd048-kube-api-access-ts8tb\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1300ca1-628e-43b9-8dfc-fcd0d197a975-iptables-alerter-script\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-cnibin\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16511250-0c7e-4b12-841b-afeba2af4bc8-tmp-dir\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhtq\" (UniqueName: \"kubernetes.io/projected/16511250-0c7e-4b12-841b-afeba2af4bc8-kube-api-access-hjhtq\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511440    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-run-ovn-kubernetes\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovnkube-config\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovnkube-script-lib\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.511591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-kubelet\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.512059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-var-lib-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.512059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-ovn\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.512059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.511615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.513442 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.513427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.515361 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.515344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:06:00.515426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.515412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-blzvj\""
Sep 17 23:06:00.515749 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.515733    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:06:00.515795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.515767    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.515838 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.515810    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:00.517983 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.517971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.520169 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.520155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.520324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.520174    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.520324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.520209    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.520324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.520209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sjvpx\""
Sep 17 23:06:00.522366 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.522347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.522847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.522831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:06:00.522847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.522843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:06:00.522932 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.522866    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7zl9\""
Sep 17 23:06:00.524202 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.524185    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:06:00.524333 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.524300    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:06:00.524394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.524331    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v4xbp\""
Sep 17 23:06:00.524394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.524345    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:06:00.539257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.539239    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fcnj6"
Sep 17 23:06:00.546796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.546781    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fcnj6"
Sep 17 23:06:00.548913 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.548882    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:00:59 +0000 UTC" deadline="2028-03-18 21:36:34.358842639 +0000 UTC"
Sep 17 23:06:00.548961 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.548913    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13150h30m33.809931991s"
Sep 17 23:06:00.607140 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.607124    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:06:00.612019 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-ovn\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-system-cni-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612054    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-hostroot\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54dd74cf-a6ef-4276-addf-456e170bc225-kubelet-config\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-ovn\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-log-socket\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-netns\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysctl-conf\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-system-cni-dir\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-log-socket\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af2fe43e-8f00-4eeb-b27d-1ff01d211599-serviceca\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-system-cni-dir\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-cni-multus\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-conf-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vr6q\" (UniqueName: \"kubernetes.io/projected/86c50601-fa82-4f6a-a9f0-a645bf3000c6-kube-api-access-2vr6q\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-etc-selinux\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-cni-bin\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-cni-bin\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-cni-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-cni-bin\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-device-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/621e2e6d-aeaa-4650-b04f-8dbba3a8e30b-agent-certs\") pod \"konnectivity-agent-sb77f\" (UID: \"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b\") " pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmg8\" (UniqueName: \"kubernetes.io/projected/e1300ca1-628e-43b9-8dfc-fcd0d197a975-kube-api-access-zlmg8\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.612651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-slash\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-cnibin\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c50601-fa82-4f6a-a9f0-a645bf3000c6-cni-binary-copy\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-kubelet\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612758    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-sys\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-slash\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-var-lib-kubelet\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgqs\" (UniqueName: \"kubernetes.io/projected/af2fe43e-8f00-4eeb-b27d-1ff01d211599-kube-api-access-dhgqs\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.612902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-systemd\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1300ca1-628e-43b9-8dfc-fcd0d197a975-iptables-alerter-script\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-run-systemd\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-cnibin\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16511250-0c7e-4b12-841b-afeba2af4bc8-tmp-dir\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhtq\" (UniqueName: \"kubernetes.io/projected/16511250-0c7e-4b12-841b-afeba2af4bc8-kube-api-access-hjhtq\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.613373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-run-ovn-kubernetes\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/647f02eb-3c4a-4b27-8d3a-9892a08754df-cnibin\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovnkube-config\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af2fe43e-8f00-4eeb-b27d-1ff01d211599-serviceca\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovnkube-script-lib\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-run-ovn-kubernetes\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54dd74cf-a6ef-4276-addf-456e170bc225-dbus\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-var-lib-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-socket-dir-parent\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-registration-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnsm\" (UniqueName: \"kubernetes.io/projected/f6d76e29-20cd-4357-800b-e3a1f8936668-kube-api-access-sdnsm\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjhn\" (UniqueName: \"kubernetes.io/projected/4404befb-c49e-440b-9f60-9b04e47390d0-kube-api-access-ggjhn\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysctl-d\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-socket-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-var-lib-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16511250-0c7e-4b12-841b-afeba2af4bc8-tmp-dir\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-modprobe-d\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-tuned\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1300ca1-628e-43b9-8dfc-fcd0d197a975-iptables-alerter-script\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hslxd\" (UniqueName: \"kubernetes.io/projected/fa83c935-2fe4-4594-a547-2d44d590beb6-kube-api-access-hslxd\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/621e2e6d-aeaa-4650-b04f-8dbba3a8e30b-konnectivity-ca\") pod \"konnectivity-agent-sb77f\" (UID: \"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b\") " pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovnkube-config\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613747    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-systemd-units\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovnkube-script-lib\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-systemd-units\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-run-netns\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-run-netns\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-etc-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-etc-openvswitch\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-kubernetes\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.614761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-systemd\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa83c935-2fe4-4594-a547-2d44d590beb6-tmp\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.613981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvjc\" (UniqueName: \"kubernetes.io/projected/647f02eb-3c4a-4b27-8d3a-9892a08754df-kube-api-access-2dvjc\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-cni-netd\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/647f02eb-3c4a-4b27-8d3a-9892a08754df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-sys-fs\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-lib-modules\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1300ca1-628e-43b9-8dfc-fcd0d197a975-host-slash\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-cni-netd\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af2fe43e-8f00-4eeb-b27d-1ff01d211599-host\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-node-log\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1300ca1-628e-43b9-8dfc-fcd0d197a975-host-slash\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af2fe43e-8f00-4eeb-b27d-1ff01d211599-host\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovn-node-metrics-cert\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-node-log\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8tb\" (UniqueName: \"kubernetes.io/projected/7e63ef48-0367-4e32-b7af-f2f23c7dd048-kube-api-access-ts8tb\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:00.615374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-run\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16511250-0c7e-4b12-841b-afeba2af4bc8-hosts-file\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-env-overrides\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-multus-certs\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-host\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16511250-0c7e-4b12-841b-afeba2af4bc8-hosts-file\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-etc-kubernetes\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614566    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614692    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-kubelet\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-k8s-cni-cncf-io\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e63ef48-0367-4e32-b7af-f2f23c7dd048-host-kubelet\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-daemon-config\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614804    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysconfig\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.615865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.614990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e63ef48-0367-4e32-b7af-f2f23c7dd048-env-overrides\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.617282 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.617265    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:00.618155 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.618138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e63ef48-0367-4e32-b7af-f2f23c7dd048-ovn-node-metrics-cert\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.618977 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.618954    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:00.618977 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.618977    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:00.619142 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.618990    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:00.619142 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.619071    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:01.119036663 +0000 UTC m=+3.592979642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:00.621156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.621132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmg8\" (UniqueName: \"kubernetes.io/projected/e1300ca1-628e-43b9-8dfc-fcd0d197a975-kube-api-access-zlmg8\") pod \"iptables-alerter-5bsf7\" (UID: \"e1300ca1-628e-43b9-8dfc-fcd0d197a975\") " pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.621426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.621410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8tb\" (UniqueName: \"kubernetes.io/projected/7e63ef48-0367-4e32-b7af-f2f23c7dd048-kube-api-access-ts8tb\") pod \"ovnkube-node-nq45x\" (UID: \"7e63ef48-0367-4e32-b7af-f2f23c7dd048\") " pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.621477 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.621437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgqs\" (UniqueName: \"kubernetes.io/projected/af2fe43e-8f00-4eeb-b27d-1ff01d211599-kube-api-access-dhgqs\") pod \"node-ca-d98wj\" (UID: \"af2fe43e-8f00-4eeb-b27d-1ff01d211599\") " pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.621477 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.621454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhtq\" (UniqueName: \"kubernetes.io/projected/16511250-0c7e-4b12-841b-afeba2af4bc8-kube-api-access-hjhtq\") pod \"node-resolver-c6w6w\" (UID: \"16511250-0c7e-4b12-841b-afeba2af4bc8\") " pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.621639 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.621623    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvjc\" (UniqueName: \"kubernetes.io/projected/647f02eb-3c4a-4b27-8d3a-9892a08754df-kube-api-access-2dvjc\") pod \"multus-additional-cni-plugins-m2bsm\" (UID: \"647f02eb-3c4a-4b27-8d3a-9892a08754df\") " pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.715889 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-sys\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.715889 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-var-lib-kubelet\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54dd74cf-a6ef-4276-addf-456e170bc225-dbus\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-socket-dir-parent\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-registration-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnsm\" (UniqueName: \"kubernetes.io/projected/f6d76e29-20cd-4357-800b-e3a1f8936668-kube-api-access-sdnsm\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-sys\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.715983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjhn\" (UniqueName: \"kubernetes.io/projected/4404befb-c49e-440b-9f60-9b04e47390d0-kube-api-access-ggjhn\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysctl-d\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-var-lib-kubelet\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-registration-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-socket-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54dd74cf-a6ef-4276-addf-456e170bc225-dbus\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.716070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-modprobe-d\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-socket-dir-parent\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-tuned\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysctl-d\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hslxd\" (UniqueName: \"kubernetes.io/projected/fa83c935-2fe4-4594-a547-2d44d590beb6-kube-api-access-hslxd\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-socket-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/621e2e6d-aeaa-4650-b04f-8dbba3a8e30b-konnectivity-ca\") pod \"konnectivity-agent-sb77f\" (UID: \"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b\") " pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.716165    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.716386    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:01.216366721 +0000 UTC m=+3.690309700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-modprobe-d\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-kubernetes\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-kubernetes\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-systemd\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa83c935-2fe4-4594-a547-2d44d590beb6-tmp\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-sys-fs\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-lib-modules\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-run\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.716627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-multus-certs\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-host\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-etc-kubernetes\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-k8s-cni-cncf-io\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-host\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-daemon-config\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-run\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysconfig\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-sys-fs\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-etc-kubernetes\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-system-cni-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-systemd\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-k8s-cni-cncf-io\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.716862    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-hostroot\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/621e2e6d-aeaa-4650-b04f-8dbba3a8e30b-konnectivity-ca\") pod \"konnectivity-agent-sb77f\" (UID: \"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b\") " pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-lib-modules\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysconfig\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.717466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-system-cni-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-hostroot\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:00.716911    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:01.216894208 +0000 UTC m=+3.690837241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-multus-certs\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54dd74cf-a6ef-4276-addf-456e170bc225-kubelet-config\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.716994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-netns\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysctl-conf\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54dd74cf-a6ef-4276-addf-456e170bc225-kubelet-config\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-cni-multus\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-conf-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-run-netns\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vr6q\" (UniqueName: \"kubernetes.io/projected/86c50601-fa82-4f6a-a9f0-a645bf3000c6-kube-api-access-2vr6q\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-etc-selinux\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-cni-multus\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-conf-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-cni-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717212    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-sysctl-conf\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.718018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-cni-bin\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-cni-bin\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-etc-selinux\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-device-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-daemon-config\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717413    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-multus-cni-dir\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6d76e29-20cd-4357-800b-e3a1f8936668-device-dir\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/621e2e6d-aeaa-4650-b04f-8dbba3a8e30b-agent-certs\") pod \"konnectivity-agent-sb77f\" (UID: \"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b\") " pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-cnibin\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c50601-fa82-4f6a-a9f0-a645bf3000c6-cni-binary-copy\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-kubelet\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-cnibin\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.717703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c50601-fa82-4f6a-a9f0-a645bf3000c6-host-var-lib-kubelet\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.718073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c50601-fa82-4f6a-a9f0-a645bf3000c6-cni-binary-copy\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.718601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.718571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa83c935-2fe4-4594-a547-2d44d590beb6-etc-tuned\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.719053 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.718627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa83c935-2fe4-4594-a547-2d44d590beb6-tmp\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.719293 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.719277    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/621e2e6d-aeaa-4650-b04f-8dbba3a8e30b-agent-certs\") pod \"konnectivity-agent-sb77f\" (UID: \"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b\") " pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.724843 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.724820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hslxd\" (UniqueName: \"kubernetes.io/projected/fa83c935-2fe4-4594-a547-2d44d590beb6-kube-api-access-hslxd\") pod \"tuned-n87zb\" (UID: \"fa83c935-2fe4-4594-a547-2d44d590beb6\") " pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.724928 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.724827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjhn\" (UniqueName: \"kubernetes.io/projected/4404befb-c49e-440b-9f60-9b04e47390d0-kube-api-access-ggjhn\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:00.725253 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.725238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnsm\" (UniqueName: \"kubernetes.io/projected/f6d76e29-20cd-4357-800b-e3a1f8936668-kube-api-access-sdnsm\") pod \"aws-ebs-csi-driver-node-x65xp\" (UID: \"f6d76e29-20cd-4357-800b-e3a1f8936668\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.725415 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.725400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vr6q\" (UniqueName: \"kubernetes.io/projected/86c50601-fa82-4f6a-a9f0-a645bf3000c6-kube-api-access-2vr6q\") pod \"multus-sxdqn\" (UID: \"86c50601-fa82-4f6a-a9f0-a645bf3000c6\") " pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.829089 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.829021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2bsm"
Sep 17 23:06:00.842983 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.842961    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c6w6w"
Sep 17 23:06:00.858231 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.858213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d98wj"
Sep 17 23:06:00.861831 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.861816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:00.876234 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.876218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bsf7"
Sep 17 23:06:00.880735 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.880707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sxdqn"
Sep 17 23:06:00.892329 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.892308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n87zb"
Sep 17 23:06:00.910880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.910859    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:00.916584 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.916524    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp"
Sep 17 23:06:00.926321 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:00.926299    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c41d7938cdbb09331b64496abe4d6c.slice/crio-063e78ec42cddc5b2bfadc0c2dae651a5b3e60ba0181b12846f1711522ded374 WatchSource:0}: Error finding container 063e78ec42cddc5b2bfadc0c2dae651a5b3e60ba0181b12846f1711522ded374: Status 404 returned error can't find the container with id 063e78ec42cddc5b2bfadc0c2dae651a5b3e60ba0181b12846f1711522ded374
Sep 17 23:06:00.926674 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:00.926656    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d76e29_20cd_4357_800b_e3a1f8936668.slice/crio-8d4cda258a6a624eb6e7a3c0dd976e0f6d4c243a4eaccfd28a90766dc936fdcc WatchSource:0}: Error finding container 8d4cda258a6a624eb6e7a3c0dd976e0f6d4c243a4eaccfd28a90766dc936fdcc: Status 404 returned error can't find the container with id 8d4cda258a6a624eb6e7a3c0dd976e0f6d4c243a4eaccfd28a90766dc936fdcc
Sep 17 23:06:00.927518 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:00.927499    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2fe43e_8f00_4eeb_b27d_1ff01d211599.slice/crio-cbac4ab9b160df50bbc2e39054ac8c6533ebbce2500c5e4b18760eb3d1e80dbb WatchSource:0}: Error finding container cbac4ab9b160df50bbc2e39054ac8c6533ebbce2500c5e4b18760eb3d1e80dbb: Status 404 returned error can't find the container with id cbac4ab9b160df50bbc2e39054ac8c6533ebbce2500c5e4b18760eb3d1e80dbb
Sep 17 23:06:00.928063 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:00.928045    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16511250_0c7e_4b12_841b_afeba2af4bc8.slice/crio-c9ea4ce0f508a4088890783d878db0c663957fb69a3b94ee24b086d8837f779f WatchSource:0}: Error finding container c9ea4ce0f508a4088890783d878db0c663957fb69a3b94ee24b086d8837f779f: Status 404 returned error can't find the container with id c9ea4ce0f508a4088890783d878db0c663957fb69a3b94ee24b086d8837f779f
Sep 17 23:06:00.928797 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:00.928779    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c50601_fa82_4f6a_a9f0_a645bf3000c6.slice/crio-5c11c15555e5a69f72b253145fc601918936fcdead5e0b1a11c3382e2c703ad4 WatchSource:0}: Error finding container 5c11c15555e5a69f72b253145fc601918936fcdead5e0b1a11c3382e2c703ad4: Status 404 returned error can't find the container with id 5c11c15555e5a69f72b253145fc601918936fcdead5e0b1a11c3382e2c703ad4
Sep 17 23:06:00.929664 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:00.929602    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa83c935_2fe4_4594_a547_2d44d590beb6.slice/crio-201219970b152593a5147ead6746a4cd3e1b2b261db20568f4b198769bf0424a WatchSource:0}: Error finding container 201219970b152593a5147ead6746a4cd3e1b2b261db20568f4b198769bf0424a: Status 404 returned error can't find the container with id 201219970b152593a5147ead6746a4cd3e1b2b261db20568f4b198769bf0424a
Sep 17 23:06:00.934540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.933413    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:06:00.982247 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:00.982223    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:01.120467 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.120413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:01.120541 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.120522    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:01.120541 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.120535    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:01.120605 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.120543    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:01.120605 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.120589    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:02.120576801 +0000 UTC m=+4.594519773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:01.202187 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:01.202167    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647f02eb_3c4a_4b27_8d3a_9892a08754df.slice/crio-8cdca20fa631d0c56af986a64b9506bfddeed555b3a831571a9df517d9b3e9a6 WatchSource:0}: Error finding container 8cdca20fa631d0c56af986a64b9506bfddeed555b3a831571a9df517d9b3e9a6: Status 404 returned error can't find the container with id 8cdca20fa631d0c56af986a64b9506bfddeed555b3a831571a9df517d9b3e9a6
Sep 17 23:06:01.221374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.221354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:01.221470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.221407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:01.221532 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.221487    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:01.221532 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.221525    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:01.221616 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.221538    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:02.221523432 +0000 UTC m=+4.695466406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:01.221616 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.221574    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:02.22155702 +0000 UTC m=+4.695499996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:01.549417 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.549377    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:00 +0000 UTC" deadline="2028-06-26 17:48:15.99728063 +0000 UTC"
Sep 17 23:06:01.549417 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.549412    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15546h42m14.447872676s"
Sep 17 23:06:01.636139 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.636109    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:01.636299 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:01.636231    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:01.655185 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.655132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp" event={"ID":"f6d76e29-20cd-4357-800b-e3a1f8936668","Type":"ContainerStarted","Data":"8d4cda258a6a624eb6e7a3c0dd976e0f6d4c243a4eaccfd28a90766dc936fdcc"}
Sep 17 23:06:01.659568 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.659509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d98wj" event={"ID":"af2fe43e-8f00-4eeb-b27d-1ff01d211599","Type":"ContainerStarted","Data":"cbac4ab9b160df50bbc2e39054ac8c6533ebbce2500c5e4b18760eb3d1e80dbb"}
Sep 17 23:06:01.661486 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.661444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal" event={"ID":"a14bb4ae145cfdc960ab64cc87036398","Type":"ContainerStarted","Data":"d608bce2c214560cc8534b6b0fcb8538934030f18ce4a0240f5ca34dacd77232"}
Sep 17 23:06:01.667876 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.667850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerStarted","Data":"8cdca20fa631d0c56af986a64b9506bfddeed555b3a831571a9df517d9b3e9a6"}
Sep 17 23:06:01.670384 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.670356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sb77f" event={"ID":"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b","Type":"ContainerStarted","Data":"3368cc4d792fd66cf711ce90ae55533d27ed93e865597bd937162debf8327d48"}
Sep 17 23:06:01.673970 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.673913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n87zb" event={"ID":"fa83c935-2fe4-4594-a547-2d44d590beb6","Type":"ContainerStarted","Data":"201219970b152593a5147ead6746a4cd3e1b2b261db20568f4b198769bf0424a"}
Sep 17 23:06:01.676160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.676129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bsf7" event={"ID":"e1300ca1-628e-43b9-8dfc-fcd0d197a975","Type":"ContainerStarted","Data":"ff8837133c9154536951e8976d2ba20e94a9321f09cf52359b041c94d98bb746"}
Sep 17 23:06:01.687918 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.687798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sxdqn" event={"ID":"86c50601-fa82-4f6a-a9f0-a645bf3000c6","Type":"ContainerStarted","Data":"5c11c15555e5a69f72b253145fc601918936fcdead5e0b1a11c3382e2c703ad4"}
Sep 17 23:06:01.692060 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.691982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"52e5ff52767c5cf888811340143c9f85b5798b1199596f20fa9d3e9de3479cad"}
Sep 17 23:06:01.698021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.697961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c6w6w" event={"ID":"16511250-0c7e-4b12-841b-afeba2af4bc8","Type":"ContainerStarted","Data":"c9ea4ce0f508a4088890783d878db0c663957fb69a3b94ee24b086d8837f779f"}
Sep 17 23:06:01.705706 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.705649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal" event={"ID":"55c41d7938cdbb09331b64496abe4d6c","Type":"ContainerStarted","Data":"063e78ec42cddc5b2bfadc0c2dae651a5b3e60ba0181b12846f1711522ded374"}
Sep 17 23:06:01.891881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:01.891798    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:02.129454 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.129417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:02.129648 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.129632    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:02.129713 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.129656    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:02.129713 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.129668    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:02.129838 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.129741    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:04.1297069 +0000 UTC m=+6.603649864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:02.230496 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.230462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:02.230674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.230533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:02.230674 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.230643    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:02.230811 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.230700    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:04.230682966 +0000 UTC m=+6.704625946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:02.230811 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.230778    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:02.230811 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.230812    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:04.230799424 +0000 UTC m=+6.704742398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:02.550317 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.550215    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:00 +0000 UTC" deadline="2028-07-05 07:18:35.869110156 +0000 UTC"
Sep 17 23:06:02.550317 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.550269    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15752h12m33.318844754s"
Sep 17 23:06:02.634230 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.632966    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:02.634230 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.633088    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:02.634230 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:02.633157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:02.634230 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:02.633222    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:03.633864 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:03.633836    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:03.634319 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:03.633990    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:04.147923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:04.147876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:04.148119 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.148086    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:04.148119 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.148109    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:04.148240 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.148123    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:04.148240 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.148182    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:08.148163887 +0000 UTC m=+10.622106862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:04.248223 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:04.248178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:04.248392 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:04.248261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:04.248392 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.248381    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:04.248517 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.248441    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:08.248422148 +0000 UTC m=+10.722365123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:04.248869 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.248847    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:04.248971 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.248903    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:08.248888674 +0000 UTC m=+10.722831639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:04.633750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:04.633686    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:04.633750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:04.633739    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:04.634251 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.633828    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:04.634251 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:04.633928    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:05.634678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:05.634644    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:05.635126 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:05.634826    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:06.633086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:06.633057    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:06.633086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:06.633081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:06.633328 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:06.633178    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:06.633328 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:06.633262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:07.633634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:07.633592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:07.634077 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:07.633761    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:08.179243 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:08.179209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:08.179461 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.179424    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:08.179461 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.179454    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:08.179461 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.179468    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:08.179650 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.179537    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:16.179516832 +0000 UTC m=+18.653459800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:08.280893 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:08.280390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:08.280893 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:08.280464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:08.280893 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.280620    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:08.280893 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.280681    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:16.280661978 +0000 UTC m=+18.754604952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:08.280893 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.280772    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:08.280893 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.280819    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:16.280805962 +0000 UTC m=+18.754748935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:08.633419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:08.633385    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:08.633616 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:08.633428    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:08.633616 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.633513    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:08.633616 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:08.633596    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:09.634734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:09.634680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:09.635161 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:09.634830    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:10.632994 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:10.632961    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:10.632994 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:10.632987    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:10.633216 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:10.633101    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:10.633280 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:10.633230    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:11.633855 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:11.633827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:11.634269 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:11.633943    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:12.633085 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:12.633045    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:12.633265 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:12.633168    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:12.633265 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:12.633201    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:12.633377 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:12.633270    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:13.633774 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:13.633741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:13.634331 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:13.633877    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:14.632899 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:14.632873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:14.633086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:14.632873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:14.633086 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:14.632987    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:14.633191 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:14.633118    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:15.632930 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:15.632901    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:15.633368 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:15.633038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:16.236232 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:16.236195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:16.236396 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.236368    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:16.236396 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.236391    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:16.236483 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.236404    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:16.236483 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.236467    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:32.236447542 +0000 UTC m=+34.710390509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:16.337198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:16.337168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:16.337367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:16.337216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:16.337367 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.337312    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:16.337367 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.337317    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:16.337367 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.337367    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:32.337350563 +0000 UTC m=+34.811293523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:16.337549 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.337381    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:32.337374483 +0000 UTC m=+34.811317443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:16.633382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:16.633301    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:16.633792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:16.633318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:16.633792 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.633439    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:16.633792 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:16.633490    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:17.633324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:17.633293    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:17.633579 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:17.633411    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:18.633490 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:18.633461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:18.633634 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:18.633559    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:18.633634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:18.633611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:18.633914 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:18.633694    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:19.635332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.634684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:19.635332 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:19.635148    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:19.756772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.756748    2559 generic.go:356] "Generic (PLEG): container finished" podID="647f02eb-3c4a-4b27-8d3a-9892a08754df" containerID="cb300bf7ed1c9ee2d9688f708085f15498f387b6b7a15ca5bae03d41a962ab77" exitCode=0
Sep 17 23:06:19.756882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.756830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerDied","Data":"cb300bf7ed1c9ee2d9688f708085f15498f387b6b7a15ca5bae03d41a962ab77"}
Sep 17 23:06:19.758332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.758281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sb77f" event={"ID":"621e2e6d-aeaa-4650-b04f-8dbba3a8e30b","Type":"ContainerStarted","Data":"597cd33174469c9810cea99dc35ecb2ded86db4f2f7786f7ca56e6403a44d94d"}
Sep 17 23:06:19.759908 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.759888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n87zb" event={"ID":"fa83c935-2fe4-4594-a547-2d44d590beb6","Type":"ContainerStarted","Data":"78303540e2c169a09fb7fb19c7d50ea392cecc850cc34bbf70b9f2f067efd4df"}
Sep 17 23:06:19.765027 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.764990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sxdqn" event={"ID":"86c50601-fa82-4f6a-a9f0-a645bf3000c6","Type":"ContainerStarted","Data":"7f554106eeb845749782ecb8417c9684b3ae6a90c57ea9c867c8c55c4a565c86"}
Sep 17 23:06:19.767912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.767892    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:06:19.768217 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.768196    2559 generic.go:356] "Generic (PLEG): container finished" podID="7e63ef48-0367-4e32-b7af-f2f23c7dd048" containerID="d2cb402735547ba6619630dbb23f19f48c88562056318af9477a1c832f621ba3" exitCode=1
Sep 17 23:06:19.768302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.768258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"52152d3f7d81921dacef5a6b4a0b4b94ab44c1c09edf07f2309a50e341e9ea87"}
Sep 17 23:06:19.768362 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.768329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"829129be5c8e55bfdadc6b186929104a3d95c37172f015a2109ef0dff6feee73"}
Sep 17 23:06:19.768409 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.768402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"3678028f717094fa8eba694f6f95da848ff60cc68a5bc550905857d5555ce828"}
Sep 17 23:06:19.768466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.768449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerDied","Data":"d2cb402735547ba6619630dbb23f19f48c88562056318af9477a1c832f621ba3"}
Sep 17 23:06:19.768523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.768489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"2704b571ef38b3c6e14eef052ce783e626ac0947b27bf0687a0f78c5614acd66"}
Sep 17 23:06:19.769804 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.769709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c6w6w" event={"ID":"16511250-0c7e-4b12-841b-afeba2af4bc8","Type":"ContainerStarted","Data":"b6dfc6f9331587421c1326ce9489b6a42ab52079e9378aa69efa3f8ec9ca840f"}
Sep 17 23:06:19.771126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.771107    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal" event={"ID":"55c41d7938cdbb09331b64496abe4d6c","Type":"ContainerStarted","Data":"f0633c6b045f8e4e2b6c8c3ffbf0c37716d2f0f374810f8d4c8c7ad25c6d6b79"}
Sep 17 23:06:19.772645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.772625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp" event={"ID":"f6d76e29-20cd-4357-800b-e3a1f8936668","Type":"ContainerStarted","Data":"9833b6b6daff3dce1c0a20b19747a432a0855e74c542fa61e01ac47a0b92d448"}
Sep 17 23:06:19.774241 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.774222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d98wj" event={"ID":"af2fe43e-8f00-4eeb-b27d-1ff01d211599","Type":"ContainerStarted","Data":"cb4cc4e18a7ecac05951a6360e5d1125213d629f2ee7da1d5e5d1a447083191a"}
Sep 17 23:06:19.775945 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.775925    2559 generic.go:356] "Generic (PLEG): container finished" podID="a14bb4ae145cfdc960ab64cc87036398" containerID="6cc774cce0fbfbaab4d13524985cb9628620d263223dd84a6c440f9cbf3ad6ba" exitCode=0
Sep 17 23:06:19.776024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.775955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal" event={"ID":"a14bb4ae145cfdc960ab64cc87036398","Type":"ContainerDied","Data":"6cc774cce0fbfbaab4d13524985cb9628620d263223dd84a6c440f9cbf3ad6ba"}
Sep 17 23:06:19.796185 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.796136    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sxdqn" podStartSLOduration=1.448449227 podStartE2EDuration="19.796120048s" totalImagesPullingTime="18.347670821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:19.795687753 +0000 UTC m=+22.269630735" watchObservedRunningTime="2026-09-17 23:06:19.796120048 +0000 UTC m=+22.270063030"
Sep 17 23:06:19.812908 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.812868    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n87zb" podStartSLOduration=1.731218738 podStartE2EDuration="19.812856569s" totalImagesPullingTime="18.081637831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:19.812389709 +0000 UTC m=+22.286332691" watchObservedRunningTime="2026-09-17 23:06:19.812856569 +0000 UTC m=+22.286799550"
Sep 17 23:06:19.826508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.826475    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sb77f" podStartSLOduration=5.691769129 podStartE2EDuration="19.826462571s" totalImagesPullingTime="14.134693442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:19.826279928 +0000 UTC m=+22.300222913" watchObservedRunningTime="2026-09-17 23:06:19.826462571 +0000 UTC m=+22.300405553"
Sep 17 23:06:19.880664 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.880617    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c6w6w" podStartSLOduration=2.798615324 podStartE2EDuration="20.88060278s" totalImagesPullingTime="18.081987456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:19.87980678 +0000 UTC m=+22.353749764" watchObservedRunningTime="2026-09-17 23:06:19.88060278 +0000 UTC m=+22.354545762"
Sep 17 23:06:19.907993 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.907959    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d98wj" podStartSLOduration=2.824831776 podStartE2EDuration="20.907944635s" totalImagesPullingTime="18.083112859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:19.894431977 +0000 UTC m=+22.368374958" watchObservedRunningTime="2026-09-17 23:06:19.907944635 +0000 UTC m=+22.381887595"
Sep 17 23:06:19.908404 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:19.908380    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-250.ec2.internal" podStartSLOduration=19.908365013 podStartE2EDuration="19.908365013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:19.908023123 +0000 UTC m=+22.381966087" watchObservedRunningTime="2026-09-17 23:06:19.908365013 +0000 UTC m=+22.382307994"
Sep 17 23:06:20.633017 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.632984    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:20.633120 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:20.633085    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:20.633120 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.633091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:20.633217 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:20.633164    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:20.639004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.638972    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:06:20.780329 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.780090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bsf7" event={"ID":"e1300ca1-628e-43b9-8dfc-fcd0d197a975","Type":"ContainerStarted","Data":"77a668a4e69e5fe70f2daeb7a9e7b6f17b1bce5660ebd11617582ba86dadcf30"}
Sep 17 23:06:20.783417 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.783398    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:06:20.783870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.783845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"36e567c264c2b0505e076539862b752a61cdc18d317fc730fa21c780cfca144f"}
Sep 17 23:06:20.785827 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.785807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp" event={"ID":"f6d76e29-20cd-4357-800b-e3a1f8936668","Type":"ContainerStarted","Data":"9c763e9d687b4d9794a554274f160240fe5ebc20d3463949bfe25075ba9e5f43"}
Sep 17 23:06:20.787796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.787747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal" event={"ID":"a14bb4ae145cfdc960ab64cc87036398","Type":"ContainerStarted","Data":"2a8ebf1b0015d798c8e3387fc6da533274c04f569514128b6423adb2fb74c002"}
Sep 17 23:06:20.809075 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.808997    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5bsf7" podStartSLOduration=3.72702399 podStartE2EDuration="21.808985401s" totalImagesPullingTime="18.081961411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:20.794892027 +0000 UTC m=+23.268835011" watchObservedRunningTime="2026-09-17 23:06:20.808985401 +0000 UTC m=+23.282928374"
Sep 17 23:06:20.809656 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:20.809626    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-250.ec2.internal" podStartSLOduration=20.809614715 podStartE2EDuration="20.809614715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:20.808972197 +0000 UTC m=+23.282915177" watchObservedRunningTime="2026-09-17 23:06:20.809614715 +0000 UTC m=+23.283557697"
Sep 17 23:06:21.599479 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:21.599385    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:06:20.638988354Z","UUID":"18011335-722a-4272-8499-469138de2b6c","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:06:21.600824 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:21.600800    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:06:21.600964 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:21.600836    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:06:21.633560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:21.633541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:21.633654 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:21.633633    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:21.791912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:21.791869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp" event={"ID":"f6d76e29-20cd-4357-800b-e3a1f8936668","Type":"ContainerStarted","Data":"4740e5dab85f5a03eff1b75e05c627b4761f7380b2a25c166d6b2ed008e2c4c2"}
Sep 17 23:06:21.810254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:21.810208    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x65xp" podStartSLOduration=2.213236342 podStartE2EDuration="21.81019308s" totalImagesPullingTime="19.596956738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:21.809790251 +0000 UTC m=+24.283733233" watchObservedRunningTime="2026-09-17 23:06:21.81019308 +0000 UTC m=+24.284136061"
Sep 17 23:06:22.633371 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:22.633337    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:22.633603 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:22.633433    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:22.633603 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:22.633479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:22.633603 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:22.633561    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:22.796892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:22.796868    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:06:22.797297 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:22.797276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"69279b4713f8b356dd05367d4c9580d72a998a963fb07f72291aeebb347fee70"}
Sep 17 23:06:23.633494 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:23.633465    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:23.633666 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:23.633581    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:24.178120 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:24.178031    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:24.178653 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:24.178631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:24.632923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:24.632894    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:24.633095 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:24.632937    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:24.633095 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:24.633010    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:24.633204 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:24.633135    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:24.802926 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:24.802895    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:24.803427 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:24.803403    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sb77f"
Sep 17 23:06:25.633599 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:25.633579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:25.633934 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:25.633741    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:25.806599 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:25.806573    2559 generic.go:356] "Generic (PLEG): container finished" podID="647f02eb-3c4a-4b27-8d3a-9892a08754df" containerID="34e044a76bafa60bbfd74112adf0c7c84bd0f0b0ab88c44b3074678d3385386a" exitCode=0
Sep 17 23:06:25.806706 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:25.806661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerDied","Data":"34e044a76bafa60bbfd74112adf0c7c84bd0f0b0ab88c44b3074678d3385386a"}
Sep 17 23:06:25.810384 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:25.810363    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:06:25.810890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:25.810865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"4244cbb4740747876a80f300938eeceed3beda64ce4a24a507e0f10738aece06"}
Sep 17 23:06:25.811297 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:25.811278    2559 scope.go:117] "RemoveContainer" containerID="d2cb402735547ba6619630dbb23f19f48c88562056318af9477a1c832f621ba3"
Sep 17 23:06:26.633634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.633594    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:26.633634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.633635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:26.634303 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:26.633750    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:26.634303 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:26.633862    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:26.816463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.816431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerStarted","Data":"f42e3d1352a1af9a52ba653d6589e3aa3eedc72e0d02605e9cb6dbd4c71ab1df"}
Sep 17 23:06:26.820542 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.820521    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:06:26.821074 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.821047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" event={"ID":"7e63ef48-0367-4e32-b7af-f2f23c7dd048","Type":"ContainerStarted","Data":"e6d23bd55510ff19e2be57e995a6ecdbd96d0bbde72b8af21cabfbaf4f4e12f4"}
Sep 17 23:06:26.821343 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.821325    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:26.821432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.821409    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:26.821491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.821468    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:26.835709 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.835686    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:26.836061 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.836044    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:06:26.863973 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:26.863933    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x" podStartSLOduration=9.748744336 podStartE2EDuration="27.863920683s" totalImagesPullingTime="18.115176347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:26.863764043 +0000 UTC m=+29.337707028" watchObservedRunningTime="2026-09-17 23:06:26.863920683 +0000 UTC m=+29.337863691"
Sep 17 23:06:27.186316 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.186288    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qpfx"]
Sep 17 23:06:27.186466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.186384    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:27.186525 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:27.186465    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:27.189801 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.189777    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kd746"]
Sep 17 23:06:27.189931 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.189873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:27.190014 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:27.189961    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:27.190475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.190453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7pb7"]
Sep 17 23:06:27.190549 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.190534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:27.190623 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:27.190606    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:27.825223 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.825142    2559 generic.go:356] "Generic (PLEG): container finished" podID="647f02eb-3c4a-4b27-8d3a-9892a08754df" containerID="f42e3d1352a1af9a52ba653d6589e3aa3eedc72e0d02605e9cb6dbd4c71ab1df" exitCode=0
Sep 17 23:06:27.825223 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.825169    2559 generic.go:356] "Generic (PLEG): container finished" podID="647f02eb-3c4a-4b27-8d3a-9892a08754df" containerID="31b610d5c95ecead273251310863779aedba842a7894a96eba342ba66586d970" exitCode=0
Sep 17 23:06:27.825757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.825222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerDied","Data":"f42e3d1352a1af9a52ba653d6589e3aa3eedc72e0d02605e9cb6dbd4c71ab1df"}
Sep 17 23:06:27.825757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:27.825270    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerDied","Data":"31b610d5c95ecead273251310863779aedba842a7894a96eba342ba66586d970"}
Sep 17 23:06:28.633671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:28.633641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:28.633806 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:28.633653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:28.633876 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:28.633805    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:28.633922 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:28.633879    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:29.633948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:29.633687    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:29.634456 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:29.634005    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:30.633018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:30.632985    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:30.633182 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:30.633098    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:30.633182 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:30.633158    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:30.633315 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:30.633293    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:31.633576 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:31.633542    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:31.634058 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:31.633668    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qpfx" podUID="54dd74cf-a6ef-4276-addf-456e170bc225"
Sep 17 23:06:32.261025 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:32.260951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:32.261179 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.261158    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:32.261219 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.261191    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:32.261219 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.261207    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ldpch for pod openshift-network-diagnostics/network-check-target-h7pb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:32.261279 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.261270    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch podName:de5746ff-f319-434a-b213-3997b9a97873 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.261251941 +0000 UTC m=+66.735194920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldpch" (UniqueName: "kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch") pod "network-check-target-h7pb7" (UID: "de5746ff-f319-434a-b213-3997b9a97873") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:32.362054 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:32.362020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:32.362201 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:32.362078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:32.362201 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.362173    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:32.362359 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.362207    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:32.362359 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.362247    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.362229037 +0000 UTC m=+66.836172002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:32.362359 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.362266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret podName:54dd74cf-a6ef-4276-addf-456e170bc225 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.362256671 +0000 UTC m=+66.836199632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret") pod "global-pull-secret-syncer-4qpfx" (UID: "54dd74cf-a6ef-4276-addf-456e170bc225") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:32.633700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:32.633630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:32.633700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:32.633653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:32.634162 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.633774    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:06:32.634162 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:32.633964    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7pb7" podUID="de5746ff-f319-434a-b213-3997b9a97873"
Sep 17 23:06:33.087501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.087477    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-250.ec2.internal" event="NodeReady"
Sep 17 23:06:33.087639 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.087626    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:06:33.119989 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.119964    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"]
Sep 17 23:06:33.134151 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.134125    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d84cd7677-wfvrz"]
Sep 17 23:06:33.134271 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.134251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.136266 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.136246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 23:06:33.136390 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.136344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 23:06:33.136390 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.136374    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v6csm\""
Sep 17 23:06:33.146758 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.146738    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"]
Sep 17 23:06:33.146838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.146769    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vnpbs"]
Sep 17 23:06:33.146838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.146808    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.149913 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.149892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:06:33.149977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.149961    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:06:33.150268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.150249    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:06:33.150381 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.150304    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4qnpb\""
Sep 17 23:06:33.162916 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.162865    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:06:33.168145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.168122    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d84cd7677-wfvrz"]
Sep 17 23:06:33.168232 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.168151    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b4vqv"]
Sep 17 23:06:33.168302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.168247    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.170332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.170306    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:06:33.170442 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.170340    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:06:33.170442 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.170368    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:06:33.170442 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.170376    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9ck9b\""
Sep 17 23:06:33.190821 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.190801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnpbs"]
Sep 17 23:06:33.190914 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.190825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4vqv"]
Sep 17 23:06:33.190959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.190926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.193108 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.193088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:06:33.193198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.193156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:06:33.193322 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.193307    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7hp2b\""
Sep 17 23:06:33.268485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpc5d\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-kube-api-access-hpc5d\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.268576 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-certificates\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.268576 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-installation-pull-secrets\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.268576 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-bound-sa-token\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.268576 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f20abf2-d57f-4040-bb02-94cd0beb80fb-tmp-dir\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.268712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdlw\" (UniqueName: \"kubernetes.io/projected/e42d7a0c-f350-4657-b80b-f024df6affb0-kube-api-access-kbdlw\") pod \"ingress-canary-vnpbs\" (UID: \"e42d7a0c-f350-4657-b80b-f024df6affb0\") " pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.268712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.268712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.268712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e42d7a0c-f350-4657-b80b-f024df6affb0-cert\") pod \"ingress-canary-vnpbs\" (UID: \"e42d7a0c-f350-4657-b80b-f024df6affb0\") " pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.268891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f20abf2-d57f-4040-bb02-94cd0beb80fb-config-volume\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.268891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.268891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-trusted-ca\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.268891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268822    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqs7\" (UniqueName: \"kubernetes.io/projected/0f20abf2-d57f-4040-bb02-94cd0beb80fb-kube-api-access-7rqs7\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.268891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-image-registry-private-configuration\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.269040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-ca-trust-extracted\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.269040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.268943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.286557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.286525    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:33.369184 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.369296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpc5d\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-kube-api-access-hpc5d\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:33.369407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-certificates\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-installation-pull-secrets\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-bound-sa-token\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369553 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f20abf2-d57f-4040-bb02-94cd0beb80fb-tmp-dir\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.369553 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdlw\" (UniqueName: \"kubernetes.io/projected/e42d7a0c-f350-4657-b80b-f024df6affb0-kube-api-access-kbdlw\") pod \"ingress-canary-vnpbs\" (UID: \"e42d7a0c-f350-4657-b80b-f024df6affb0\") " pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.369553 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.369553 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e42d7a0c-f350-4657-b80b-f024df6affb0-cert\") pod \"ingress-canary-vnpbs\" (UID: \"e42d7a0c-f350-4657-b80b-f024df6affb0\") " pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f20abf2-d57f-4040-bb02-94cd0beb80fb-config-volume\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-trusted-ca\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqs7\" (UniqueName: \"kubernetes.io/projected/0f20abf2-d57f-4040-bb02-94cd0beb80fb-kube-api-access-7rqs7\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-image-registry-private-configuration\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.369768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-ca-trust-extracted\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.369797    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.369813    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.369861    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:06:33.869844628 +0000 UTC m=+36.343787590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f20abf2-d57f-4040-bb02-94cd0beb80fb-tmp-dir\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.369981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-certificates\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.370057    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:06:33.370116 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.370103    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:33.870087589 +0000 UTC m=+36.344030550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:06:33.370478 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.370202    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:06:33.370478 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.370241    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:06:33.870226976 +0000 UTC m=+36.344169942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:06:33.370478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.370336    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-ca-trust-extracted\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.370586 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.370484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f20abf2-d57f-4040-bb02-94cd0beb80fb-config-volume\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.370916 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.370890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-trusted-ca\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.373658 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.373640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-image-registry-private-configuration\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.373772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.373638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-installation-pull-secrets\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.373772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.373764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e42d7a0c-f350-4657-b80b-f024df6affb0-cert\") pod \"ingress-canary-vnpbs\" (UID: \"e42d7a0c-f350-4657-b80b-f024df6affb0\") " pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.377315 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.377287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqs7\" (UniqueName: \"kubernetes.io/projected/0f20abf2-d57f-4040-bb02-94cd0beb80fb-kube-api-access-7rqs7\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.377648 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.377630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-bound-sa-token\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.377696 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.377646    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdlw\" (UniqueName: \"kubernetes.io/projected/e42d7a0c-f350-4657-b80b-f024df6affb0-kube-api-access-kbdlw\") pod \"ingress-canary-vnpbs\" (UID: \"e42d7a0c-f350-4657-b80b-f024df6affb0\") " pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.378332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.378309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpc5d\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-kube-api-access-hpc5d\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.476109 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.476085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnpbs"
Sep 17 23:06:33.633130 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.633094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:06:33.635808 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.635781    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:06:33.783768 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.783668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:33.846183 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.846017    2559 generic.go:356] "Generic (PLEG): container finished" podID="647f02eb-3c4a-4b27-8d3a-9892a08754df" containerID="6b8195b9571666ad9ba28dd445d6534f80b7e6c5be57f132d69383935c9710df" exitCode=0
Sep 17 23:06:33.846342 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.846089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerDied","Data":"6b8195b9571666ad9ba28dd445d6534f80b7e6c5be57f132d69383935c9710df"}
Sep 17 23:06:33.872479 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.872454    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnpbs"]
Sep 17 23:06:33.872822 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.872795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:33.872944 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.872852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:33.872944 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.872882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:33.873083 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873017    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:06:33.873083 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873070    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:06:34.873053535 +0000 UTC m=+37.346996509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:06:33.873193 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873116    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:06:33.873193 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:34.873165284 +0000 UTC m=+37.347108250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:06:33.873303 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873244    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:06:33.873303 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873260    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:06:33.873303 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:33.873304    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:06:34.873290827 +0000 UTC m=+37.347233790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:06:33.877235 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:06:33.877215    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42d7a0c_f350_4657_b80b_f024df6affb0.slice/crio-46cc54f4981b482e43f601b3c7677180fbaac9eae2805517f9cdbc5936b29661 WatchSource:0}: Error finding container 46cc54f4981b482e43f601b3c7677180fbaac9eae2805517f9cdbc5936b29661: Status 404 returned error can't find the container with id 46cc54f4981b482e43f601b3c7677180fbaac9eae2805517f9cdbc5936b29661
Sep 17 23:06:33.886420 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:33.886397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:34.633363 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.633328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:06:34.633568 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.633533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:06:34.636021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.635995    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:06:34.636021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.636008    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:06:34.640014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.637438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:06:34.640014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.637512    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twn2q\""
Sep 17 23:06:34.640014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.637653    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hrj9x\""
Sep 17 23:06:34.850614 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.850572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnpbs" event={"ID":"e42d7a0c-f350-4657-b80b-f024df6affb0","Type":"ContainerStarted","Data":"46cc54f4981b482e43f601b3c7677180fbaac9eae2805517f9cdbc5936b29661"}
Sep 17 23:06:34.854545 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.854511    2559 generic.go:356] "Generic (PLEG): container finished" podID="647f02eb-3c4a-4b27-8d3a-9892a08754df" containerID="8c2ff027a1b72af54b93aad96ae227b05d86461f1227ad28454e05fd7a41b142" exitCode=0
Sep 17 23:06:34.854671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.854578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerDied","Data":"8c2ff027a1b72af54b93aad96ae227b05d86461f1227ad28454e05fd7a41b142"}
Sep 17 23:06:34.882695 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.882454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:34.882695 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.882517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:34.882695 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:34.882547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:34.882924 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.882744    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:06:34.882924 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.882800    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:06:36.882782653 +0000 UTC m=+39.356725616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:06:34.883237 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.883147    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:06:34.883237 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.883213    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:36.883196775 +0000 UTC m=+39.357139741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:06:34.883237 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.883150    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:06:34.883237 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.883238    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:06:34.883448 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:34.883274    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:06:36.883264065 +0000 UTC m=+39.357207031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:06:35.154934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:35.154863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:35.214104 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:35.214071    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:35.860961 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:35.860930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" event={"ID":"647f02eb-3c4a-4b27-8d3a-9892a08754df","Type":"ContainerStarted","Data":"d009b464d3f1e8ec51862bb957c908a96f0344230d6a0e769868e073c9a19934"}
Sep 17 23:06:35.881832 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:35.881778    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m2bsm" podStartSLOduration=6.7909476269999995 podStartE2EDuration="36.881761977s" totalImagesPullingTime="30.09081435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:35.880663189 +0000 UTC m=+38.354606171" watchObservedRunningTime="2026-09-17 23:06:35.881761977 +0000 UTC m=+38.355704959"
Sep 17 23:06:36.001158 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.001123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:36.100556 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.100521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:36.776337 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.776313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:36.879351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.879323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:36.899733 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.899693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:36.899826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.899775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:36.899826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:36.899804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:36.899904 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899823    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:06:36.899904 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899872    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:40.899858997 +0000 UTC m=+43.373801963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:06:36.899904 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899895    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:06:36.900012 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899912    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:06:36.900012 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899951    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:06:40.899938365 +0000 UTC m=+43.373881326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:06:36.900012 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899950    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:06:36.900012 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:36.899989    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:06:40.899981969 +0000 UTC m=+43.373924930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:06:37.495450 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:37.495421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:37.594038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:37.594011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:37.867082 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:37.866999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnpbs" event={"ID":"e42d7a0c-f350-4657-b80b-f024df6affb0","Type":"ContainerStarted","Data":"78e2d15ce1b43fc76924382071fc19a411b97c8e5666f6f67ab137f53fc0363c"}
Sep 17 23:06:37.883393 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:37.883338    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vnpbs" podStartSLOduration=1.9885509620000001 podStartE2EDuration="4.88332385s" totalImagesPullingTime="2.894772888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:37.883288304 +0000 UTC m=+40.357231286" watchObservedRunningTime="2026-09-17 23:06:37.88332385 +0000 UTC m=+40.357266825"
Sep 17 23:06:40.933009 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:40.932971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:40.933026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:40.933044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933129    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933154    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933195    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:06:48.933181968 +0000 UTC m=+51.407124929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933207    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:48.933200972 +0000 UTC m=+51.407143934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933132    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933218    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:06:40.933401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:40.933238    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:06:48.93323216 +0000 UTC m=+51.407175121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:06:48.991458 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:48.991415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:48.991467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:48.991488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991569    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991592    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991637    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.991623632 +0000 UTC m=+67.465566594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991672    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.991658549 +0000 UTC m=+67.465601509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991594    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991690    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:06:48.991848 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:06:48.991747    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.991737845 +0000 UTC m=+67.465680809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:06:58.841424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:06:58.841387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nq45x"
Sep 17 23:07:04.295689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.295649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:07:04.298503 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.298485    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:07:04.307998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.307980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:07:04.319800 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.319773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpch\" (UniqueName: \"kubernetes.io/projected/de5746ff-f319-434a-b213-3997b9a97873-kube-api-access-ldpch\") pod \"network-check-target-h7pb7\" (UID: \"de5746ff-f319-434a-b213-3997b9a97873\") " pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:07:04.350757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.350732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twn2q\""
Sep 17 23:07:04.359712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.359691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:07:04.396949 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.396921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:07:04.397077 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.397024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:07:04.399417 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.399396    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:07:04.399487 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.399453    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:07:04.408308 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:04.408283    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:07:04.408425 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:04.408378    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:08:08.408357758 +0000 UTC m=+130.882300724 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : secret "metrics-daemon-secret" not found
Sep 17 23:07:04.410226 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.410206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54dd74cf-a6ef-4276-addf-456e170bc225-original-pull-secret\") pod \"global-pull-secret-syncer-4qpfx\" (UID: \"54dd74cf-a6ef-4276-addf-456e170bc225\") " pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:07:04.545776 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.545665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qpfx"
Sep 17 23:07:04.651922 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.651891    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7pb7"]
Sep 17 23:07:04.655321 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:07:04.655293    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5746ff_f319_434a_b213_3997b9a97873.slice/crio-382dad9ba77ff4e3c5a66bdf2d74caa30f422bdd454f10359a81ad5486f40b57 WatchSource:0}: Error finding container 382dad9ba77ff4e3c5a66bdf2d74caa30f422bdd454f10359a81ad5486f40b57: Status 404 returned error can't find the container with id 382dad9ba77ff4e3c5a66bdf2d74caa30f422bdd454f10359a81ad5486f40b57
Sep 17 23:07:04.809499 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.809410    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qpfx"]
Sep 17 23:07:04.813497 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:07:04.813472    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dd74cf_a6ef_4276_addf_456e170bc225.slice/crio-190a712c4d7714d73e3bfe06f749fc5979174a943f7adf8f6b56cb67901f8b26 WatchSource:0}: Error finding container 190a712c4d7714d73e3bfe06f749fc5979174a943f7adf8f6b56cb67901f8b26: Status 404 returned error can't find the container with id 190a712c4d7714d73e3bfe06f749fc5979174a943f7adf8f6b56cb67901f8b26
Sep 17 23:07:04.934135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.934083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qpfx" event={"ID":"54dd74cf-a6ef-4276-addf-456e170bc225","Type":"ContainerStarted","Data":"190a712c4d7714d73e3bfe06f749fc5979174a943f7adf8f6b56cb67901f8b26"}
Sep 17 23:07:04.935269 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:04.935244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7pb7" event={"ID":"de5746ff-f319-434a-b213-3997b9a97873","Type":"ContainerStarted","Data":"382dad9ba77ff4e3c5a66bdf2d74caa30f422bdd454f10359a81ad5486f40b57"}
Sep 17 23:07:05.002362 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:05.002310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:07:05.002362 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:05.002369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:05.002391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002480    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002539    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002547    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002563    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002566    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:37.002543919 +0000 UTC m=+99.476486881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002591    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:07:37.002579616 +0000 UTC m=+99.476522576 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:07:05.002628 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:05.002614    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:07:37.002596979 +0000 UTC m=+99.476539943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:07:09.950463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:09.950420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qpfx" event={"ID":"54dd74cf-a6ef-4276-addf-456e170bc225","Type":"ContainerStarted","Data":"c5c343601cc6ac670ad4d62860d03f4545c3a0f4c4cb42e6e5bce249e896f35c"}
Sep 17 23:07:09.951948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:09.951921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7pb7" event={"ID":"de5746ff-f319-434a-b213-3997b9a97873","Type":"ContainerStarted","Data":"ba67f3d32f362f4558042585964423af77b396ab8d768312defe09a335f3b18f"}
Sep 17 23:07:09.952085 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:09.952068    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:07:09.968862 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:09.968820    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4qpfx" podStartSLOduration=65.536043545 podStartE2EDuration="1m9.968805959s" totalImagesPullingTime="4.432762414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:09.968029066 +0000 UTC m=+72.441972050" watchObservedRunningTime="2026-09-17 23:07:09.968805959 +0000 UTC m=+72.442748941"
Sep 17 23:07:09.986201 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:09.986145    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h7pb7" podStartSLOduration=66.399589999 podStartE2EDuration="1m10.986132188s" totalImagesPullingTime="4.586542189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:09.984540334 +0000 UTC m=+72.458483317" watchObservedRunningTime="2026-09-17 23:07:09.986132188 +0000 UTC m=+72.460075171"
Sep 17 23:07:37.040516 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:37.040433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:07:37.040516 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:37.040474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:07:37.040516 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:37.040494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040581    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040582    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040597    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040615    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040637    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:08:41.040624652 +0000 UTC m=+163.514567613 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040649    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:08:41.040643636 +0000 UTC m=+163.514586597 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:07:37.041016 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:07:37.040678    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:08:41.040660103 +0000 UTC m=+163.514603085 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:07:40.956274 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:07:40.956243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h7pb7"
Sep 17 23:08:08.466620 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:08.466576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:08:08.467115 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:08.466747    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:08:08.467115 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:08.466822    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:10.466806009 +0000 UTC m=+252.940748970 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : secret "metrics-daemon-secret" not found
Sep 17 23:08:36.144000 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:36.143941    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx" podUID="ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1"
Sep 17 23:08:36.157344 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:36.157322    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"
Sep 17 23:08:36.164175 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:36.164152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:08:36.164278 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:36.164153    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:08:36.198643 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:36.198621    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b4vqv" podUID="0f20abf2-d57f-4040-bb02-94cd0beb80fb"
Sep 17 23:08:37.654435 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:37.654389    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:08:41.087351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:41.087308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:41.087363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:41.087435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087474    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087489    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d84cd7677-wfvrz: secret "image-registry-tls" not found
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087522    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087552    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls podName:86ddb0e4-cd95-4fde-be49-09bffe7d9d1a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.087533486 +0000 UTC m=+285.561476449 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls") pod "image-registry-5d84cd7677-wfvrz" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a") : secret "image-registry-tls" not found
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087580    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls podName:0f20abf2-d57f-4040-bb02-94cd0beb80fb nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.087567097 +0000 UTC m=+285.561510059 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls") pod "dns-default-b4vqv" (UID: "0f20abf2-d57f-4040-bb02-94cd0beb80fb") : secret "dns-default-metrics-tls" not found
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087584    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 23:08:41.087990 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:08:41.087648    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert podName:ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.08763313 +0000 UTC m=+285.561576096 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-qhhvx" (UID: "ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1") : secret "networking-console-plugin-cert" not found
Sep 17 23:08:49.634888 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:49.634806    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:08:51.633688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:08:51.633652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:10:09.352769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.352735    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"]
Sep 17 23:10:09.355524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.355503    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77f8ffff48-x67qv"]
Sep 17 23:10:09.355634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.355617    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.357914 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.357893    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 23:10:09.358039 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.357915    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:09.358259 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.358233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 23:10:09.358370 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.358289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.358370 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.358237    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-k8f5r"]
Sep 17 23:10:09.358498 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.358370    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:09.358498 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.358376    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rz559\""
Sep 17 23:10:09.361019 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361001    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.361432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361413    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 23:10:09.361432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361427    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 23:10:09.361693 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 23:10:09.361791 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361698    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 23:10:09.361791 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361706    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 23:10:09.361878 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.361827    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 23:10:09.362180 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.362166    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5htjv\""
Sep 17 23:10:09.364080 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.364064    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q82db\""
Sep 17 23:10:09.365052 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.365034    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 23:10:09.365228 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.365209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 23:10:09.365280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.365237    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:09.365314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.365242    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:09.369031 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.369009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"]
Sep 17 23:10:09.369970 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.369949    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 23:10:09.374391 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.374370    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77f8ffff48-x67qv"]
Sep 17 23:10:09.379617 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.379600    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-k8f5r"]
Sep 17 23:10:09.449593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.449564    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"]
Sep 17 23:10:09.452302 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.452288    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-x5n54"]
Sep 17 23:10:09.452404 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.452389    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"
Sep 17 23:10:09.454793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.454777    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dwzs2\""
Sep 17 23:10:09.455330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.455312    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"]
Sep 17 23:10:09.455487 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.455470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.457712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.457694    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:10:09.457821 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.457767    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 23:10:09.457821 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.457803    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:10:09.457974 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.457959    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 23:10:09.458093 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.458071    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f7fj7\""
Sep 17 23:10:09.458199 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.458098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.461727 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.461704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:10:09.461826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.461750    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 23:10:09.461826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.461710    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:10:09.461972 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.461959    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 23:10:09.462143 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.462128    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l62k7\""
Sep 17 23:10:09.464582 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.464563    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 23:10:09.464942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.464923    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"]
Sep 17 23:10:09.466163 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.466142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"]
Sep 17 23:10:09.467372 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.467351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-x5n54"]
Sep 17 23:10:09.499051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fdf7271-1621-4a9f-8b5c-78fed07d772f-serving-cert\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.499207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.499207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmn8\" (UniqueName: \"kubernetes.io/projected/06ef5388-8c10-4002-a3ae-1ddb404a54e7-kube-api-access-dsmn8\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.499207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnl75\" (UniqueName: \"kubernetes.io/projected/7fdf7271-1621-4a9f-8b5c-78fed07d772f-kube-api-access-lnl75\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.499207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-default-certificate\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.499351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06ef5388-8c10-4002-a3ae-1ddb404a54e7-config\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.499351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499280    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.499351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06ef5388-8c10-4002-a3ae-1ddb404a54e7-serving-cert\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.499351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06ef5388-8c10-4002-a3ae-1ddb404a54e7-trusted-ca\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.499473 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fdf7271-1621-4a9f-8b5c-78fed07d772f-config\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.499473 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-stats-auth\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.499473 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.499410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4bk\" (UniqueName: \"kubernetes.io/projected/ceed8fa5-da35-4ace-a55b-428695ec6af8-kube-api-access-nl4bk\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.599750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.599714    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe98b390-a870-4ae3-9656-cb7d1dc56c29-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.599851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.599764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fdf7271-1621-4a9f-8b5c-78fed07d772f-serving-cert\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.599851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.599781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmn8\" (UniqueName: \"kubernetes.io/projected/06ef5388-8c10-4002-a3ae-1ddb404a54e7-kube-api-access-dsmn8\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.599851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.599795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06ef5388-8c10-4002-a3ae-1ddb404a54e7-config\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.599970 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.599947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fdf7271-1621-4a9f-8b5c-78fed07d772f-config\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.600012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.599983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-stats-auth\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.600012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4bk\" (UniqueName: \"kubernetes.io/projected/ceed8fa5-da35-4ace-a55b-428695ec6af8-kube-api-access-nl4bk\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.600117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbl5\" (UniqueName: \"kubernetes.io/projected/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-kube-api-access-zlbl5\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.600117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnl75\" (UniqueName: \"kubernetes.io/projected/7fdf7271-1621-4a9f-8b5c-78fed07d772f-kube-api-access-lnl75\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.600234 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.600234 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fe98b390-a870-4ae3-9656-cb7d1dc56c29-snapshots\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.600234 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600223    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.600401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe98b390-a870-4ae3-9656-cb7d1dc56c29-tmp\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.600401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600279    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtvh\" (UniqueName: \"kubernetes.io/projected/5948eefd-2a23-400e-8931-fbf2de25e5bb-kube-api-access-gxtvh\") pod \"network-check-source-7cbc45fddc-v4qgc\" (UID: \"5948eefd-2a23-400e-8931-fbf2de25e5bb\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"
Sep 17 23:10:09.600401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:09.600308    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:10:09.600401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.600401 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:09.600357    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:10.100340649 +0000 UTC m=+252.574283622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : secret "router-metrics-certs-default" not found
Sep 17 23:10:09.600401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe98b390-a870-4ae3-9656-cb7d1dc56c29-serving-cert\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06ef5388-8c10-4002-a3ae-1ddb404a54e7-config\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe98b390-a870-4ae3-9656-cb7d1dc56c29-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:09.600536    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:10.100516257 +0000 UTC m=+252.574459222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fdf7271-1621-4a9f-8b5c-78fed07d772f-config\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vrc\" (UniqueName: \"kubernetes.io/projected/fe98b390-a870-4ae3-9656-cb7d1dc56c29-kube-api-access-j8vrc\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-default-certificate\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.600674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06ef5388-8c10-4002-a3ae-1ddb404a54e7-serving-cert\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.600977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.600681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06ef5388-8c10-4002-a3ae-1ddb404a54e7-trusted-ca\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.601621 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.601598    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06ef5388-8c10-4002-a3ae-1ddb404a54e7-trusted-ca\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.602290 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.602266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fdf7271-1621-4a9f-8b5c-78fed07d772f-serving-cert\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.602614 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.602596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-stats-auth\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.602964 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.602923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-default-certificate\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.603097 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.603084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06ef5388-8c10-4002-a3ae-1ddb404a54e7-serving-cert\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.607946 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.607923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnl75\" (UniqueName: \"kubernetes.io/projected/7fdf7271-1621-4a9f-8b5c-78fed07d772f-kube-api-access-lnl75\") pod \"service-ca-operator-57cdc4b768-2v7x9\" (UID: \"7fdf7271-1621-4a9f-8b5c-78fed07d772f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.608087 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.607926    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4bk\" (UniqueName: \"kubernetes.io/projected/ceed8fa5-da35-4ace-a55b-428695ec6af8-kube-api-access-nl4bk\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:09.608150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.608115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmn8\" (UniqueName: \"kubernetes.io/projected/06ef5388-8c10-4002-a3ae-1ddb404a54e7-kube-api-access-dsmn8\") pod \"console-operator-fd4bb46c-k8f5r\" (UID: \"06ef5388-8c10-4002-a3ae-1ddb404a54e7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.666581 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.666559    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"
Sep 17 23:10:09.678607 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.678587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:09.701541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.701674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe98b390-a870-4ae3-9656-cb7d1dc56c29-tmp\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.701674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtvh\" (UniqueName: \"kubernetes.io/projected/5948eefd-2a23-400e-8931-fbf2de25e5bb-kube-api-access-gxtvh\") pod \"network-check-source-7cbc45fddc-v4qgc\" (UID: \"5948eefd-2a23-400e-8931-fbf2de25e5bb\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"
Sep 17 23:10:09.701674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.701870 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:09.701665    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:09.701870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe98b390-a870-4ae3-9656-cb7d1dc56c29-serving-cert\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.701870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe98b390-a870-4ae3-9656-cb7d1dc56c29-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.701870 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:09.701763    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls podName:7ea26fcd-6335-471b-8e55-07d32f9b6ff0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:10.201746832 +0000 UTC m=+252.675689793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xxsct" (UID: "7ea26fcd-6335-471b-8e55-07d32f9b6ff0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:09.701870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vrc\" (UniqueName: \"kubernetes.io/projected/fe98b390-a870-4ae3-9656-cb7d1dc56c29-kube-api-access-j8vrc\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.702125 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe98b390-a870-4ae3-9656-cb7d1dc56c29-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.702125 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbl5\" (UniqueName: \"kubernetes.io/projected/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-kube-api-access-zlbl5\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.702125 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.701956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fe98b390-a870-4ae3-9656-cb7d1dc56c29-snapshots\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.702483 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.702417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.702642 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.702577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fe98b390-a870-4ae3-9656-cb7d1dc56c29-snapshots\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.702746 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.702681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe98b390-a870-4ae3-9656-cb7d1dc56c29-tmp\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.702835 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.702811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe98b390-a870-4ae3-9656-cb7d1dc56c29-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.703469 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.703442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe98b390-a870-4ae3-9656-cb7d1dc56c29-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.704062 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.704041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe98b390-a870-4ae3-9656-cb7d1dc56c29-serving-cert\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.711371 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.711344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtvh\" (UniqueName: \"kubernetes.io/projected/5948eefd-2a23-400e-8931-fbf2de25e5bb-kube-api-access-gxtvh\") pod \"network-check-source-7cbc45fddc-v4qgc\" (UID: \"5948eefd-2a23-400e-8931-fbf2de25e5bb\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"
Sep 17 23:10:09.711602 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.711581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vrc\" (UniqueName: \"kubernetes.io/projected/fe98b390-a870-4ae3-9656-cb7d1dc56c29-kube-api-access-j8vrc\") pod \"insights-operator-6f9c644dd9-x5n54\" (UID: \"fe98b390-a870-4ae3-9656-cb7d1dc56c29\") " pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.712106 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.712084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbl5\" (UniqueName: \"kubernetes.io/projected/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-kube-api-access-zlbl5\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:09.761788 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.761599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"
Sep 17 23:10:09.769810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.769792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54"
Sep 17 23:10:09.934033 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.933957    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9"]
Sep 17 23:10:09.936924 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:09.936898    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdf7271_1621_4a9f_8b5c_78fed07d772f.slice/crio-c559c20db201bef7b28f3ecd336b8e331eb68622a7881576db05176efc2a4d6b WatchSource:0}: Error finding container c559c20db201bef7b28f3ecd336b8e331eb68622a7881576db05176efc2a4d6b: Status 404 returned error can't find the container with id c559c20db201bef7b28f3ecd336b8e331eb68622a7881576db05176efc2a4d6b
Sep 17 23:10:09.958588 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:09.958541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-k8f5r"]
Sep 17 23:10:09.964810 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:09.964782    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ef5388_8c10_4002_a3ae_1ddb404a54e7.slice/crio-44886ef7773c8ee82e11410d87bb8ca8d4f1417de4f09154a15be4f48c2a475f WatchSource:0}: Error finding container 44886ef7773c8ee82e11410d87bb8ca8d4f1417de4f09154a15be4f48c2a475f: Status 404 returned error can't find the container with id 44886ef7773c8ee82e11410d87bb8ca8d4f1417de4f09154a15be4f48c2a475f
Sep 17 23:10:10.030027 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.030004    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc"]
Sep 17 23:10:10.032400 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:10.032383    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5948eefd_2a23_400e_8931_fbf2de25e5bb.slice/crio-7f135f702306ba86fbe5b651720b9c5ad8cb604c04a97c2e9aedb9fb2ea52983 WatchSource:0}: Error finding container 7f135f702306ba86fbe5b651720b9c5ad8cb604c04a97c2e9aedb9fb2ea52983: Status 404 returned error can't find the container with id 7f135f702306ba86fbe5b651720b9c5ad8cb604c04a97c2e9aedb9fb2ea52983
Sep 17 23:10:10.039474 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.039450    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-x5n54"]
Sep 17 23:10:10.041441 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:10.041422    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe98b390_a870_4ae3_9656_cb7d1dc56c29.slice/crio-d32d79109d8fd0445e8469601b2dc0355b16c863c197f8c50df62ebeda6d92c0 WatchSource:0}: Error finding container d32d79109d8fd0445e8469601b2dc0355b16c863c197f8c50df62ebeda6d92c0: Status 404 returned error can't find the container with id d32d79109d8fd0445e8469601b2dc0355b16c863c197f8c50df62ebeda6d92c0
Sep 17 23:10:10.105862 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.105832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:10.105973 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.105909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:10.106031 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.105987    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:10:10.106086 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.106059    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:11.106037717 +0000 UTC m=+253.579980678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : secret "router-metrics-certs-default" not found
Sep 17 23:10:10.106086 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.106078    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:11.106067924 +0000 UTC m=+253.580010892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:10:10.207282 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.207248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:10.207436 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.207393    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:10.207487 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.207454    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls podName:7ea26fcd-6335-471b-8e55-07d32f9b6ff0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:11.207439293 +0000 UTC m=+253.681382254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xxsct" (UID: "7ea26fcd-6335-471b-8e55-07d32f9b6ff0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:10.389546 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.389515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9" event={"ID":"7fdf7271-1621-4a9f-8b5c-78fed07d772f","Type":"ContainerStarted","Data":"c559c20db201bef7b28f3ecd336b8e331eb68622a7881576db05176efc2a4d6b"}
Sep 17 23:10:10.390642 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.390618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54" event={"ID":"fe98b390-a870-4ae3-9656-cb7d1dc56c29","Type":"ContainerStarted","Data":"d32d79109d8fd0445e8469601b2dc0355b16c863c197f8c50df62ebeda6d92c0"}
Sep 17 23:10:10.392094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.392066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc" event={"ID":"5948eefd-2a23-400e-8931-fbf2de25e5bb","Type":"ContainerStarted","Data":"11dc69f09c58a463c2699f9381bf24ec02bb4ccb5e1b3f23961f133466e274e7"}
Sep 17 23:10:10.392195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.392108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc" event={"ID":"5948eefd-2a23-400e-8931-fbf2de25e5bb","Type":"ContainerStarted","Data":"7f135f702306ba86fbe5b651720b9c5ad8cb604c04a97c2e9aedb9fb2ea52983"}
Sep 17 23:10:10.393208 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.393186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" event={"ID":"06ef5388-8c10-4002-a3ae-1ddb404a54e7","Type":"ContainerStarted","Data":"44886ef7773c8ee82e11410d87bb8ca8d4f1417de4f09154a15be4f48c2a475f"}
Sep 17 23:10:10.406668 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.406625    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-v4qgc" podStartSLOduration=1.406611922 podStartE2EDuration="1.406611922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:10.406209711 +0000 UTC m=+252.880152699" watchObservedRunningTime="2026-09-17 23:10:10.406611922 +0000 UTC m=+252.880554905"
Sep 17 23:10:10.510181 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:10.510124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:10:10.510293 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.510254    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:10:10.510355 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:10.510308    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs podName:4404befb-c49e-440b-9f60-9b04e47390d0 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:12.510294935 +0000 UTC m=+374.984237896 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs") pod "network-metrics-daemon-kd746" (UID: "4404befb-c49e-440b-9f60-9b04e47390d0") : secret "metrics-daemon-secret" not found
Sep 17 23:10:11.118092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:11.117373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:11.118092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:11.117449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:11.118092 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:11.117607    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:13.117586738 +0000 UTC m=+255.591529699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:10:11.118092 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:11.117696    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:10:11.118092 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:11.117750    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:13.117738946 +0000 UTC m=+255.591681911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : secret "router-metrics-certs-default" not found
Sep 17 23:10:11.218557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:11.218508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:11.218775 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:11.218750    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:11.218861 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:11.218817    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls podName:7ea26fcd-6335-471b-8e55-07d32f9b6ff0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:13.218797812 +0000 UTC m=+255.692740779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xxsct" (UID: "7ea26fcd-6335-471b-8e55-07d32f9b6ff0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:13.134657 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.134557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:13.134657 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.134631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:13.135123 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:13.134761    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:10:13.135123 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:13.134850    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:17.134824348 +0000 UTC m=+259.608767310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:10:13.135123 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:13.134883    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:17.134870398 +0000 UTC m=+259.608813359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : secret "router-metrics-certs-default" not found
Sep 17 23:10:13.236077 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.236052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:13.236185 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:13.236170    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:13.236242 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:13.236234    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls podName:7ea26fcd-6335-471b-8e55-07d32f9b6ff0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:17.23621588 +0000 UTC m=+259.710158857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xxsct" (UID: "7ea26fcd-6335-471b-8e55-07d32f9b6ff0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:13.403839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.403773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9" event={"ID":"7fdf7271-1621-4a9f-8b5c-78fed07d772f","Type":"ContainerStarted","Data":"4b28b62a95a3ec7e29995f78e6e7186dedae7c68d4d652d34f263db8c62df909"}
Sep 17 23:10:13.405396 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.405372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54" event={"ID":"fe98b390-a870-4ae3-9656-cb7d1dc56c29","Type":"ContainerStarted","Data":"229df4b7524bfde22aad1b1a8b84b0f9a3b9c4443769cf62d35d7d87c1f4c7dd"}
Sep 17 23:10:13.407047 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.407030    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/0.log"
Sep 17 23:10:13.407139 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.407067    2559 generic.go:356] "Generic (PLEG): container finished" podID="06ef5388-8c10-4002-a3ae-1ddb404a54e7" containerID="1e0c9eec0b30bbcccf56e255af652f19fb13b7bd28da133b6532ba9a3ffa4374" exitCode=255
Sep 17 23:10:13.407139 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.407121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" event={"ID":"06ef5388-8c10-4002-a3ae-1ddb404a54e7","Type":"ContainerDied","Data":"1e0c9eec0b30bbcccf56e255af652f19fb13b7bd28da133b6532ba9a3ffa4374"}
Sep 17 23:10:13.407336 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.407322    2559 scope.go:117] "RemoveContainer" containerID="1e0c9eec0b30bbcccf56e255af652f19fb13b7bd28da133b6532ba9a3ffa4374"
Sep 17 23:10:13.418162 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.418120    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9" podStartSLOduration=1.152756565 podStartE2EDuration="4.418103398s" totalImagesPullingTime="3.265346833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:13.417841415 +0000 UTC m=+255.891784403" watchObservedRunningTime="2026-09-17 23:10:13.418103398 +0000 UTC m=+255.892046382"
Sep 17 23:10:13.450934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:13.450877    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54" podStartSLOduration=1.283675649 podStartE2EDuration="4.450857611s" totalImagesPullingTime="3.167181962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:13.448424223 +0000 UTC m=+255.922367206" watchObservedRunningTime="2026-09-17 23:10:13.450857611 +0000 UTC m=+255.924800595"
Sep 17 23:10:14.412629 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.412603    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:10:14.413057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.413036    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/0.log"
Sep 17 23:10:14.413124 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.413083    2559 generic.go:356] "Generic (PLEG): container finished" podID="06ef5388-8c10-4002-a3ae-1ddb404a54e7" containerID="fca17a4ac3de4cc48d1414d01953ddec542683a1c931bc69c911d116df79d1ca" exitCode=255
Sep 17 23:10:14.413200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.413173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" event={"ID":"06ef5388-8c10-4002-a3ae-1ddb404a54e7","Type":"ContainerDied","Data":"fca17a4ac3de4cc48d1414d01953ddec542683a1c931bc69c911d116df79d1ca"}
Sep 17 23:10:14.413244 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.413235    2559 scope.go:117] "RemoveContainer" containerID="1e0c9eec0b30bbcccf56e255af652f19fb13b7bd28da133b6532ba9a3ffa4374"
Sep 17 23:10:14.413547 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.413528    2559 scope.go:117] "RemoveContainer" containerID="fca17a4ac3de4cc48d1414d01953ddec542683a1c931bc69c911d116df79d1ca"
Sep 17 23:10:14.413764 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:14.413715    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-k8f5r_openshift-console-operator(06ef5388-8c10-4002-a3ae-1ddb404a54e7)\"" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" podUID="06ef5388-8c10-4002-a3ae-1ddb404a54e7"
Sep 17 23:10:14.608100 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.608028    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-58qvv"]
Sep 17 23:10:14.612606 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.612587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.614806 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.614783    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:10:14.614922 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.614813    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qmjfd\""
Sep 17 23:10:14.614922 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.614785    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:10:14.623308 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.623285    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-58qvv"]
Sep 17 23:10:14.748659 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.748627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvqq\" (UniqueName: \"kubernetes.io/projected/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-api-access-4rvqq\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.748816 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.748683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.748816 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.748744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-crio-socket\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.748816 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.748775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-data-volume\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.748954 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.748830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850164 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvqq\" (UniqueName: \"kubernetes.io/projected/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-api-access-4rvqq\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-crio-socket\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-data-volume\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850430 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850430 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:14.850278    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:10:14.850430 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-crio-socket\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.850430 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:14.850398    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:15.350376404 +0000 UTC m=+257.824319384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : configmap "kube-rbac-proxy" not found
Sep 17 23:10:14.850430 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:14.850402    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:14.850659 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:14.850482    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:15.350463799 +0000 UTC m=+257.824406776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:14.850659 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.850553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-data-volume\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:14.857822 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:14.857792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvqq\" (UniqueName: \"kubernetes.io/projected/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-api-access-4rvqq\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:15.353828 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:15.353794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:15.353997 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:15.353842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:15.353997 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:15.353924    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:10:15.353997 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:15.353956    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:15.354174 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:15.354003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:16.353980362 +0000 UTC m=+258.827923337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : configmap "kube-rbac-proxy" not found
Sep 17 23:10:15.354174 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:15.354021    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:16.354013278 +0000 UTC m=+258.827956240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:15.418388 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:15.418367    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:10:15.418784 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:15.418684    2559 scope.go:117] "RemoveContainer" containerID="fca17a4ac3de4cc48d1414d01953ddec542683a1c931bc69c911d116df79d1ca"
Sep 17 23:10:15.418859 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:15.418839    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-k8f5r_openshift-console-operator(06ef5388-8c10-4002-a3ae-1ddb404a54e7)\"" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" podUID="06ef5388-8c10-4002-a3ae-1ddb404a54e7"
Sep 17 23:10:16.360806 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:16.360756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:16.360962 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:16.360904    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:16.361014 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:16.360973    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:18.360956281 +0000 UTC m=+260.834899245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:16.361014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:16.361003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:16.361115 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:16.361090    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:10:16.361155 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:16.361132    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:18.361121461 +0000 UTC m=+260.835064435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : configmap "kube-rbac-proxy" not found
Sep 17 23:10:16.380529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:16.380503    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c6w6w_16511250-0c7e-4b12-841b-afeba2af4bc8/dns-node-resolver/0.log"
Sep 17 23:10:17.168710 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:17.168671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:17.169120 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:17.168750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:17.169120 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:17.168823    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:10:17.169120 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:17.168894    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:25.168878641 +0000 UTC m=+267.642821602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : secret "router-metrics-certs-default" not found
Sep 17 23:10:17.169120 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:17.168920    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:25.16890482 +0000 UTC m=+267.642847781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:10:17.269160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:17.269133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:17.269300 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:17.269281    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:17.269366 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:17.269356    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls podName:7ea26fcd-6335-471b-8e55-07d32f9b6ff0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:25.269338811 +0000 UTC m=+267.743281774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xxsct" (UID: "7ea26fcd-6335-471b-8e55-07d32f9b6ff0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:17.581999 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:17.581970    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d98wj_af2fe43e-8f00-4eeb-b27d-1ff01d211599/node-ca/0.log"
Sep 17 23:10:18.378067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:18.378032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:18.378485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:18.378080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:18.378485 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:18.378142    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:10:18.378485 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:18.378209    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:18.378485 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:18.378229    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.378207283 +0000 UTC m=+264.852150251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : configmap "kube-rbac-proxy" not found
Sep 17 23:10:18.378485 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:18.378250    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.378239165 +0000 UTC m=+264.852182127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:18.381310 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:18.381289    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnpbs_e42d7a0c-f350-4657-b80b-f024df6affb0/serve-healthcheck-canary/0.log"
Sep 17 23:10:19.679572 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:19.679500    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:19.679572 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:19.679550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:19.679965 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:19.679860    2559 scope.go:117] "RemoveContainer" containerID="fca17a4ac3de4cc48d1414d01953ddec542683a1c931bc69c911d116df79d1ca"
Sep 17 23:10:19.680028 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:19.680009    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-k8f5r_openshift-console-operator(06ef5388-8c10-4002-a3ae-1ddb404a54e7)\"" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" podUID="06ef5388-8c10-4002-a3ae-1ddb404a54e7"
Sep 17 23:10:22.406884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:22.406839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:22.407342 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:22.406898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:22.407342 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:22.406980    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:10:22.407342 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:22.407001    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:22.407342 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:22.407055    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:30.407042714 +0000 UTC m=+272.880985675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:22.407342 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:22.407082    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:30.40706134 +0000 UTC m=+272.881004321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : configmap "kube-rbac-proxy" not found
Sep 17 23:10:25.230103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:25.230063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:25.230103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:25.230115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:25.230483 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:25.230206    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:10:25.230483 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:25.230245    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.23023135 +0000 UTC m=+283.704174312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : configmap references non-existent config key: service-ca.crt
Sep 17 23:10:25.230483 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:25.230265    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs podName:ceed8fa5-da35-4ace-a55b-428695ec6af8 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.230251594 +0000 UTC m=+283.704194554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs") pod "router-default-77f8ffff48-x67qv" (UID: "ceed8fa5-da35-4ace-a55b-428695ec6af8") : secret "router-metrics-certs-default" not found
Sep 17 23:10:25.330713 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:25.330684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:25.330851 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:25.330816    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:25.330888 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:25.330862    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls podName:7ea26fcd-6335-471b-8e55-07d32f9b6ff0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.330850534 +0000 UTC m=+283.804793495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xxsct" (UID: "7ea26fcd-6335-471b-8e55-07d32f9b6ff0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:10:30.470970 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:30.470937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:30.471325 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:30.471042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:30.471325 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:30.471077    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:30.471325 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:30.471144    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls podName:88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c nodeName:}" failed. No retries permitted until 2026-09-17 23:10:46.471129881 +0000 UTC m=+288.945072841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-58qvv" (UID: "88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:10:30.471523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:30.471506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:32.633057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:32.633032    2559 scope.go:117] "RemoveContainer" containerID="fca17a4ac3de4cc48d1414d01953ddec542683a1c931bc69c911d116df79d1ca"
Sep 17 23:10:33.473252 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:33.473228    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:10:33.473399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:33.473274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" event={"ID":"06ef5388-8c10-4002-a3ae-1ddb404a54e7","Type":"ContainerStarted","Data":"3ce6257011aea4f7b794cfb597fabd095a8633671d5cd06d2a8a4f67e49b6bee"}
Sep 17 23:10:33.473528 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:33.473509    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:33.478014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:33.477993    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r"
Sep 17 23:10:33.492446 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:33.492408    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-k8f5r" podStartSLOduration=21.251079179 podStartE2EDuration="24.492396997s" totalImagesPullingTime="3.241317818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:33.492076314 +0000 UTC m=+275.966019297" watchObservedRunningTime="2026-09-17 23:10:33.492396997 +0000 UTC m=+275.966340018"
Sep 17 23:10:39.165236 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:39.165200    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"
Sep 17 23:10:39.165236 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:39.165218    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx" podUID="ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1"
Sep 17 23:10:39.491570 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:39.491543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:10:39.491731 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:39.491546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:10:41.250789 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.250751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:41.251220 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.250848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:41.251987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.251967    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed8fa5-da35-4ace-a55b-428695ec6af8-service-ca-bundle\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:41.253049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.253028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceed8fa5-da35-4ace-a55b-428695ec6af8-metrics-certs\") pod \"router-default-77f8ffff48-x67qv\" (UID: \"ceed8fa5-da35-4ace-a55b-428695ec6af8\") " pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:41.351508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.351469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:41.353643 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.353622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea26fcd-6335-471b-8e55-07d32f9b6ff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xxsct\" (UID: \"7ea26fcd-6335-471b-8e55-07d32f9b6ff0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:41.472792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.472767    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:41.575287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.575246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"
Sep 17 23:10:41.745305 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.745275    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77f8ffff48-x67qv"]
Sep 17 23:10:41.749565 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:41.749541    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceed8fa5_da35_4ace_a55b_428695ec6af8.slice/crio-11221c60b20cb3de206bfa5d40a50f68b5387af9a4af2d89ce2e42d4e1d796a1 WatchSource:0}: Error finding container 11221c60b20cb3de206bfa5d40a50f68b5387af9a4af2d89ce2e42d4e1d796a1: Status 404 returned error can't find the container with id 11221c60b20cb3de206bfa5d40a50f68b5387af9a4af2d89ce2e42d4e1d796a1
Sep 17 23:10:41.847932 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:41.847904    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct"]
Sep 17 23:10:41.850646 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:41.850612    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea26fcd_6335_471b_8e55_07d32f9b6ff0.slice/crio-d2bd631d120031d27e0869313ca3bf7b1649e1cad0b87a1528d54915c28a7796 WatchSource:0}: Error finding container d2bd631d120031d27e0869313ca3bf7b1649e1cad0b87a1528d54915c28a7796: Status 404 returned error can't find the container with id d2bd631d120031d27e0869313ca3bf7b1649e1cad0b87a1528d54915c28a7796
Sep 17 23:10:42.503586 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:42.503545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77f8ffff48-x67qv" event={"ID":"ceed8fa5-da35-4ace-a55b-428695ec6af8","Type":"ContainerStarted","Data":"312f06ffe0ce729cc3a62cda27a100d03ed61f262e7c32ec334e8301d9a35b59"}
Sep 17 23:10:42.504080 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:42.503609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77f8ffff48-x67qv" event={"ID":"ceed8fa5-da35-4ace-a55b-428695ec6af8","Type":"ContainerStarted","Data":"11221c60b20cb3de206bfa5d40a50f68b5387af9a4af2d89ce2e42d4e1d796a1"}
Sep 17 23:10:42.505200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:42.505162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct" event={"ID":"7ea26fcd-6335-471b-8e55-07d32f9b6ff0","Type":"ContainerStarted","Data":"d2bd631d120031d27e0869313ca3bf7b1649e1cad0b87a1528d54915c28a7796"}
Sep 17 23:10:42.523983 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:42.523938    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77f8ffff48-x67qv" podStartSLOduration=33.523922792 podStartE2EDuration="33.523922792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:42.522495831 +0000 UTC m=+284.996438814" watchObservedRunningTime="2026-09-17 23:10:42.523922792 +0000 UTC m=+284.997865777"
Sep 17 23:10:43.167351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.167317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:10:43.167525 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.167366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:10:43.167525 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.167407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:10:43.170057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.170026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f20abf2-d57f-4040-bb02-94cd0beb80fb-metrics-tls\") pod \"dns-default-b4vqv\" (UID: \"0f20abf2-d57f-4040-bb02-94cd0beb80fb\") " pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:10:43.170188 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.170170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"image-registry-5d84cd7677-wfvrz\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") " pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:10:43.170244 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.170182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-qhhvx\" (UID: \"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:10:43.337379 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.337346    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7hp2b\""
Sep 17 23:10:43.345387 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.345367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:10:43.394817 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.394793    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v6csm\""
Sep 17 23:10:43.394945 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.394793    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4qnpb\""
Sep 17 23:10:43.402844 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.402826    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"
Sep 17 23:10:43.402934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.402855    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:10:43.473418 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.473374    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:43.476634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.476214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:43.508796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.508765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:43.510055 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.510026    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77f8ffff48-x67qv"
Sep 17 23:10:43.644308 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.644171    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4vqv"]
Sep 17 23:10:43.646815 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:43.646788    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f20abf2_d57f_4040_bb02_94cd0beb80fb.slice/crio-e3face67121901a4a5c574adf5d68338acdbaf4631e5dc07db953d9b7dab49ec WatchSource:0}: Error finding container e3face67121901a4a5c574adf5d68338acdbaf4631e5dc07db953d9b7dab49ec: Status 404 returned error can't find the container with id e3face67121901a4a5c574adf5d68338acdbaf4631e5dc07db953d9b7dab49ec
Sep 17 23:10:43.695776 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.695705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d84cd7677-wfvrz"]
Sep 17 23:10:43.698578 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.698524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx"]
Sep 17 23:10:43.699613 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:43.699464    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ddb0e4_cd95_4fde_be49_09bffe7d9d1a.slice/crio-1cdbce587e52aabbf85547e42f0a5a368733a00c410ec8ca68c3d6f993eb0e20 WatchSource:0}: Error finding container 1cdbce587e52aabbf85547e42f0a5a368733a00c410ec8ca68c3d6f993eb0e20: Status 404 returned error can't find the container with id 1cdbce587e52aabbf85547e42f0a5a368733a00c410ec8ca68c3d6f993eb0e20
Sep 17 23:10:43.702013 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:43.701989    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4cf7f0_9ac6_4e3a_85b7_d5892828e9d1.slice/crio-513960937bcd572615cb4ff5d5de5f62067dc9529329f2a305f30033f78c94df WatchSource:0}: Error finding container 513960937bcd572615cb4ff5d5de5f62067dc9529329f2a305f30033f78c94df: Status 404 returned error can't find the container with id 513960937bcd572615cb4ff5d5de5f62067dc9529329f2a305f30033f78c94df
Sep 17 23:10:43.951193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.951165    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"]
Sep 17 23:10:43.958688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.958665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:43.961143 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.961121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 23:10:43.961143 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.961144    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cdhfm\""
Sep 17 23:10:43.963012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:43.962959    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"]
Sep 17 23:10:44.075783 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.075743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a948559c-749d-4abd-8979-b49885584561-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-c5c66\" (UID: \"a948559c-749d-4abd-8979-b49885584561\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:44.177061 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.177025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a948559c-749d-4abd-8979-b49885584561-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-c5c66\" (UID: \"a948559c-749d-4abd-8979-b49885584561\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:44.177241 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:44.177183    2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 23:10:44.177311 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:44.177297    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a948559c-749d-4abd-8979-b49885584561-tls-certificates podName:a948559c-749d-4abd-8979-b49885584561 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:44.677274568 +0000 UTC m=+287.151217529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a948559c-749d-4abd-8979-b49885584561-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-c5c66" (UID: "a948559c-749d-4abd-8979-b49885584561") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 23:10:44.515166 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.515044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct" event={"ID":"7ea26fcd-6335-471b-8e55-07d32f9b6ff0","Type":"ContainerStarted","Data":"889fcd8e9853de9006aab023cd8e487a47d8d376f83308b2faade8338a505f59"}
Sep 17 23:10:44.516567 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.516530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx" event={"ID":"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1","Type":"ContainerStarted","Data":"513960937bcd572615cb4ff5d5de5f62067dc9529329f2a305f30033f78c94df"}
Sep 17 23:10:44.518575 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.518500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" event={"ID":"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a","Type":"ContainerStarted","Data":"4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136"}
Sep 17 23:10:44.518575 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.518562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" event={"ID":"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a","Type":"ContainerStarted","Data":"1cdbce587e52aabbf85547e42f0a5a368733a00c410ec8ca68c3d6f993eb0e20"}
Sep 17 23:10:44.518781 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.518592    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:10:44.520098 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.520070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4vqv" event={"ID":"0f20abf2-d57f-4040-bb02-94cd0beb80fb","Type":"ContainerStarted","Data":"e3face67121901a4a5c574adf5d68338acdbaf4631e5dc07db953d9b7dab49ec"}
Sep 17 23:10:44.550785 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.550712    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xxsct" podStartSLOduration=33.978485723 podStartE2EDuration="35.550695696s" totalImagesPullingTime="1.572209973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:44.530159324 +0000 UTC m=+287.004102309" watchObservedRunningTime="2026-09-17 23:10:44.550695696 +0000 UTC m=+287.024638679"
Sep 17 23:10:44.552023 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.551980    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" podStartSLOduration=285.551965192 podStartE2EDuration="4m45.551965192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:44.550206562 +0000 UTC m=+287.024149545" watchObservedRunningTime="2026-09-17 23:10:44.551965192 +0000 UTC m=+287.025908176"
Sep 17 23:10:44.682073 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.682033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a948559c-749d-4abd-8979-b49885584561-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-c5c66\" (UID: \"a948559c-749d-4abd-8979-b49885584561\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:44.685076 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.685023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a948559c-749d-4abd-8979-b49885584561-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-c5c66\" (UID: \"a948559c-749d-4abd-8979-b49885584561\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:44.869121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:44.869040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:45.176115 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:45.176032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"]
Sep 17 23:10:45.419820 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:45.419787    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda948559c_749d_4abd_8979_b49885584561.slice/crio-83b58cc43567c13abf1d657fec0be55c7dcad870887f03f091a297d021ac5c6e WatchSource:0}: Error finding container 83b58cc43567c13abf1d657fec0be55c7dcad870887f03f091a297d021ac5c6e: Status 404 returned error can't find the container with id 83b58cc43567c13abf1d657fec0be55c7dcad870887f03f091a297d021ac5c6e
Sep 17 23:10:45.524642 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:45.524604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66" event={"ID":"a948559c-749d-4abd-8979-b49885584561","Type":"ContainerStarted","Data":"83b58cc43567c13abf1d657fec0be55c7dcad870887f03f091a297d021ac5c6e"}
Sep 17 23:10:45.526119 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:45.526076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx" event={"ID":"ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1","Type":"ContainerStarted","Data":"27fba5a69342e8657ae4e84a40fb9691b863ba792877fde611f8e3bce3bb478b"}
Sep 17 23:10:46.497781 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.497750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:46.499971 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.499941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58qvv\" (UID: \"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c\") " pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:46.531530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.531500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4vqv" event={"ID":"0f20abf2-d57f-4040-bb02-94cd0beb80fb","Type":"ContainerStarted","Data":"04b2ac5011232a91c4ca586b19e6882d2ed34ba3ee4491315d1a151f746884b7"}
Sep 17 23:10:46.531858 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.531554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4vqv" event={"ID":"0f20abf2-d57f-4040-bb02-94cd0beb80fb","Type":"ContainerStarted","Data":"75fce5dceb3687494816d036de8f8585cebee385c595b5e681acd3889d7c2afb"}
Sep 17 23:10:46.531858 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.531784    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:10:46.543071 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.543009    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-qhhvx" podStartSLOduration=273.289294622 podStartE2EDuration="4m34.542990314s" totalImagesPullingTime="1.253695692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:45.539744253 +0000 UTC m=+288.013687230" watchObservedRunningTime="2026-09-17 23:10:46.542990314 +0000 UTC m=+289.016933298"
Sep 17 23:10:46.543429 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.543409    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bd6cb97f7-nhwx8"]
Sep 17 23:10:46.551543 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.551502    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b4vqv" podStartSLOduration=251.73308193 podStartE2EDuration="4m13.551487541s" totalImagesPullingTime="1.818405611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:46.55021649 +0000 UTC m=+289.024159475" watchObservedRunningTime="2026-09-17 23:10:46.551487541 +0000 UTC m=+289.025430523"
Sep 17 23:10:46.566923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.566888    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd6cb97f7-nhwx8"]
Sep 17 23:10:46.567024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.567007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.570205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570041    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:10:46.570205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570040    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:10:46.570205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570086    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:10:46.570205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570118    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:10:46.570205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:10:46.570205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570195    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:10:46.570540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570454    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rk242\""
Sep 17 23:10:46.570540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.570483    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:10:46.700559 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.700525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrvm\" (UniqueName: \"kubernetes.io/projected/21910f22-9119-4e58-a78f-ec5086cf2141-kube-api-access-wdrvm\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.700708 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.700581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-service-ca\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.700708 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.700641    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-serving-cert\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.700708 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.700672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-oauth-config\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.700708 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.700702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-console-config\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.700926 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.700792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-oauth-serving-cert\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.721559 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.721533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-58qvv"
Sep 17 23:10:46.801408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.801352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-oauth-serving-cert\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.801408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.801389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrvm\" (UniqueName: \"kubernetes.io/projected/21910f22-9119-4e58-a78f-ec5086cf2141-kube-api-access-wdrvm\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.801534 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.801423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-service-ca\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.801534 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.801444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-serving-cert\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.801534 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.801525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-oauth-config\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.801635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.801556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-console-config\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.802135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.802104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-service-ca\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.802228 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.802190    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-oauth-serving-cert\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.802288 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.802249    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-console-config\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.803833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.803814    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-serving-cert\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.803904 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.803853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-oauth-config\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.808642 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.808622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrvm\" (UniqueName: \"kubernetes.io/projected/21910f22-9119-4e58-a78f-ec5086cf2141-kube-api-access-wdrvm\") pod \"console-bd6cb97f7-nhwx8\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") " pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:46.877239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:46.877213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:47.115832 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.115808    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-58qvv"]
Sep 17 23:10:47.167791 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.167766    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd6cb97f7-nhwx8"]
Sep 17 23:10:47.227462 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:47.227437    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e83fe1_e0b8_4ee6_baa5_aa8e74f7797c.slice/crio-f965c7afd8470db2fa392118b92fe812c497a2ba120127b6fd5cf2b0dd223ea3 WatchSource:0}: Error finding container f965c7afd8470db2fa392118b92fe812c497a2ba120127b6fd5cf2b0dd223ea3: Status 404 returned error can't find the container with id f965c7afd8470db2fa392118b92fe812c497a2ba120127b6fd5cf2b0dd223ea3
Sep 17 23:10:47.228122 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:47.228101    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21910f22_9119_4e58_a78f_ec5086cf2141.slice/crio-5ad78b8c0a5c4c6804566a60c45ccd6774be1cc83cf4b314f190250a34642cca WatchSource:0}: Error finding container 5ad78b8c0a5c4c6804566a60c45ccd6774be1cc83cf4b314f190250a34642cca: Status 404 returned error can't find the container with id 5ad78b8c0a5c4c6804566a60c45ccd6774be1cc83cf4b314f190250a34642cca
Sep 17 23:10:47.535891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.535846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd6cb97f7-nhwx8" event={"ID":"21910f22-9119-4e58-a78f-ec5086cf2141","Type":"ContainerStarted","Data":"5ad78b8c0a5c4c6804566a60c45ccd6774be1cc83cf4b314f190250a34642cca"}
Sep 17 23:10:47.537309 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.537284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58qvv" event={"ID":"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c","Type":"ContainerStarted","Data":"b0a10f4acd9534bef508626d09a2efdf1647aee7a0f91c47bd5bc172735cc790"}
Sep 17 23:10:47.537419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.537343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58qvv" event={"ID":"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c","Type":"ContainerStarted","Data":"f965c7afd8470db2fa392118b92fe812c497a2ba120127b6fd5cf2b0dd223ea3"}
Sep 17 23:10:47.538783 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.538757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66" event={"ID":"a948559c-749d-4abd-8979-b49885584561","Type":"ContainerStarted","Data":"dede9b609f7a1b08bb0951f9648606f45ede40f013dc720a6a5d90ba4f526843"}
Sep 17 23:10:47.539128 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.539110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:47.543406 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.543384    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66"
Sep 17 23:10:47.553140 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:47.553097    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-c5c66" podStartSLOduration=2.715528704 podStartE2EDuration="4.553083029s" totalImagesPullingTime="1.837554325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:47.552037973 +0000 UTC m=+290.025980957" watchObservedRunningTime="2026-09-17 23:10:47.553083029 +0000 UTC m=+290.027026041"
Sep 17 23:10:48.006972 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.006932    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"]
Sep 17 23:10:48.010359 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.010333    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.012663 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.012637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 23:10:48.013566 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.013465    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:10:48.013753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.013713    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 23:10:48.013840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.013745    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8lkm4\""
Sep 17 23:10:48.020398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.020371    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"]
Sep 17 23:10:48.114171 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.114144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f549r\" (UniqueName: \"kubernetes.io/projected/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-kube-api-access-f549r\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.114282 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.114218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.114282 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.114259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.114402 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.114318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.216459 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.215774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f549r\" (UniqueName: \"kubernetes.io/projected/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-kube-api-access-f549r\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.216459 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.215851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.216459 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.215894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.216459 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.215961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.217041 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.217015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.219195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.219160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.221355 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.221328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.226590 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.226548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f549r\" (UniqueName: \"kubernetes.io/projected/42a1c7f2-ca1b-4c18-ae98-cf9e40209076-kube-api-access-f549r\") pod \"prometheus-operator-7bd5d56d6-rlbkf\" (UID: \"42a1c7f2-ca1b-4c18-ae98-cf9e40209076\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.325974 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.325682    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"
Sep 17 23:10:48.549138 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.549098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58qvv" event={"ID":"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c","Type":"ContainerStarted","Data":"1e9fbab74a46fb14f13092d80fdb7db3dc723f917c803d0f72fad7913c9a1b1d"}
Sep 17 23:10:48.612851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:48.612735    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf"]
Sep 17 23:10:49.300524 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:49.300488    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a1c7f2_ca1b_4c18_ae98_cf9e40209076.slice/crio-82ecb69c889ca275c834e38fb2a746993042efe305ff9a43ba7d2c3a5bd75f19 WatchSource:0}: Error finding container 82ecb69c889ca275c834e38fb2a746993042efe305ff9a43ba7d2c3a5bd75f19: Status 404 returned error can't find the container with id 82ecb69c889ca275c834e38fb2a746993042efe305ff9a43ba7d2c3a5bd75f19
Sep 17 23:10:49.559043 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:49.558957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf" event={"ID":"42a1c7f2-ca1b-4c18-ae98-cf9e40209076","Type":"ContainerStarted","Data":"82ecb69c889ca275c834e38fb2a746993042efe305ff9a43ba7d2c3a5bd75f19"}
Sep 17 23:10:52.522281 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.522245    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b8c7db9df-dz6pz"]
Sep 17 23:10:52.525314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.525296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.532884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.532863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:10:52.537268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.537249    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c7db9df-dz6pz"]
Sep 17 23:10:52.573922 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.573896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd6cb97f7-nhwx8" event={"ID":"21910f22-9119-4e58-a78f-ec5086cf2141","Type":"ContainerStarted","Data":"7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48"}
Sep 17 23:10:52.576093 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.576064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58qvv" event={"ID":"88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c","Type":"ContainerStarted","Data":"d8a6a7683de80dbef68fa45161a459acff22a008489866183ef76f41293570ad"}
Sep 17 23:10:52.577806 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.577778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf" event={"ID":"42a1c7f2-ca1b-4c18-ae98-cf9e40209076","Type":"ContainerStarted","Data":"e5058aa69ff7f3612371a0c6ce5fe6f8af434bc8aba661be55709d53415b4912"}
Sep 17 23:10:52.577876 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.577831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf" event={"ID":"42a1c7f2-ca1b-4c18-ae98-cf9e40209076","Type":"ContainerStarted","Data":"a376fd149ea20f4d86d802eacc298e96b5a7549350682e6b04d3f7eded16c5e7"}
Sep 17 23:10:52.589238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.589198    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bd6cb97f7-nhwx8" podStartSLOduration=1.924135919 podStartE2EDuration="6.589185614s" totalImagesPullingTime="4.665049695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:52.588624931 +0000 UTC m=+295.062567915" watchObservedRunningTime="2026-09-17 23:10:52.589185614 +0000 UTC m=+295.063128597"
Sep 17 23:10:52.603093 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.603051    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-rlbkf" podStartSLOduration=2.933270418 podStartE2EDuration="5.603035938s" totalImagesPullingTime="2.66976552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:52.603024862 +0000 UTC m=+295.076967846" watchObservedRunningTime="2026-09-17 23:10:52.603035938 +0000 UTC m=+295.076978923"
Sep 17 23:10:52.619565 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.619528    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-58qvv" podStartSLOduration=34.495124949 podStartE2EDuration="38.619516411s" totalImagesPullingTime="4.124391462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:52.618771247 +0000 UTC m=+295.092714231" watchObservedRunningTime="2026-09-17 23:10:52.619516411 +0000 UTC m=+295.093459395"
Sep 17 23:10:52.657341 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-console-config\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.657432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-service-ca\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.657432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-trusted-ca-bundle\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.657545 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-oauth-serving-cert\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.657596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-serving-cert\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.657637 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-oauth-config\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.657679 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.657639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhvj\" (UniqueName: \"kubernetes.io/projected/68f92918-bba6-4d9f-ae33-554b3d51652d-kube-api-access-gfhvj\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.758935 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.758913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-oauth-serving-cert\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.758946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-serving-cert\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.758963    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-oauth-config\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.758990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhvj\" (UniqueName: \"kubernetes.io/projected/68f92918-bba6-4d9f-ae33-554b3d51652d-kube-api-access-gfhvj\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.759048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-console-config\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.759073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-service-ca\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.759134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-trusted-ca-bundle\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759713 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.759688    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-oauth-serving-cert\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759713 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.759706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-console-config\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.759882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.759697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-service-ca\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.760050 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.760026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-trusted-ca-bundle\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.761426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.761406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-serving-cert\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.761499 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.761458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-oauth-config\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.766346 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.766323    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhvj\" (UniqueName: \"kubernetes.io/projected/68f92918-bba6-4d9f-ae33-554b3d51652d-kube-api-access-gfhvj\") pod \"console-b8c7db9df-dz6pz\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") " pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:52.834396 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:52.834347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:10:53.104890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:53.104823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c7db9df-dz6pz"]
Sep 17 23:10:53.107684 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:53.107658    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f92918_bba6_4d9f_ae33_554b3d51652d.slice/crio-4718a07be187085aa5d0539555879e789d24f206243312cfaf8c8e68cd9cf3c6 WatchSource:0}: Error finding container 4718a07be187085aa5d0539555879e789d24f206243312cfaf8c8e68cd9cf3c6: Status 404 returned error can't find the container with id 4718a07be187085aa5d0539555879e789d24f206243312cfaf8c8e68cd9cf3c6
Sep 17 23:10:53.582732 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:53.582694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c7db9df-dz6pz" event={"ID":"68f92918-bba6-4d9f-ae33-554b3d51652d","Type":"ContainerStarted","Data":"073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8"}
Sep 17 23:10:53.583166 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:53.582770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c7db9df-dz6pz" event={"ID":"68f92918-bba6-4d9f-ae33-554b3d51652d","Type":"ContainerStarted","Data":"4718a07be187085aa5d0539555879e789d24f206243312cfaf8c8e68cd9cf3c6"}
Sep 17 23:10:53.600215 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:53.600175    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b8c7db9df-dz6pz" podStartSLOduration=1.600162963 podStartE2EDuration="1.600162963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:53.598967939 +0000 UTC m=+296.072910922" watchObservedRunningTime="2026-09-17 23:10:53.600162963 +0000 UTC m=+296.074105947"
Sep 17 23:10:54.341146 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.341115    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"]
Sep 17 23:10:54.344694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.344666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.346980 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.346955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:10:54.347098 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.346962    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cq2p4\""
Sep 17 23:10:54.347098 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.347054    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 23:10:54.356488 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.356461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"]
Sep 17 23:10:54.366127 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.366106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jqgpc"]
Sep 17 23:10:54.371029 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.371009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.374738 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.374709    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:10:54.374866 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.374846    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n6f65\""
Sep 17 23:10:54.375027 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.375010    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:10:54.375105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.375091    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:10:54.472300 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-root\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchsp\" (UniqueName: \"kubernetes.io/projected/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-kube-api-access-lchsp\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.472467 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-textfile\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472467 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.472559 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472614 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-wtmp\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472614 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d353d6cb-d949-4cdf-b425-0b1df26f10ea-metrics-client-ca\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.472734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-tls\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpt79\" (UniqueName: \"kubernetes.io/projected/d353d6cb-d949-4cdf-b425-0b1df26f10ea-kube-api-access-bpt79\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-sys\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.472851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.472813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.573403 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchsp\" (UniqueName: \"kubernetes.io/projected/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-kube-api-access-lchsp\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.573503 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-textfile\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573503 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.573576 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-wtmp\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d353d6cb-d949-4cdf-b425-0b1df26f10ea-metrics-client-ca\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-textfile\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.573795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-tls\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573893 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-wtmp\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.573893 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:54.573822    2559 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 23:10:54.573893 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:54.573858    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:10:54.574024 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:54.573898    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-tls podName:f47eb749-c30c-4f90-aa6c-1bd247fca6b3 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.073877281 +0000 UTC m=+297.547820261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-s4jt4" (UID: "f47eb749-c30c-4f90-aa6c-1bd247fca6b3") : secret "openshift-state-metrics-tls" not found
Sep 17 23:10:54.574024 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:54.573916    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-tls podName:d353d6cb-d949-4cdf-b425-0b1df26f10ea nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.073906752 +0000 UTC m=+297.547849715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-tls") pod "node-exporter-jqgpc" (UID: "d353d6cb-d949-4cdf-b425-0b1df26f10ea") : secret "node-exporter-tls" not found
Sep 17 23:10:54.574024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpt79\" (UniqueName: \"kubernetes.io/projected/d353d6cb-d949-4cdf-b425-0b1df26f10ea-kube-api-access-bpt79\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-sys\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.573995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.574321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-sys\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-root\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d353d6cb-d949-4cdf-b425-0b1df26f10ea-root\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d353d6cb-d949-4cdf-b425-0b1df26f10ea-metrics-client-ca\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574321    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.574463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.574315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.576303 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.576283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.576385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.576372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.582522 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.582504    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchsp\" (UniqueName: \"kubernetes.io/projected/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-kube-api-access-lchsp\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:54.583058 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:54.583041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpt79\" (UniqueName: \"kubernetes.io/projected/d353d6cb-d949-4cdf-b425-0b1df26f10ea-kube-api-access-bpt79\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:54.634661 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:10:54.634606    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kd746" podUID="4404befb-c49e-440b-9f60-9b04e47390d0"
Sep 17 23:10:55.078260 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.078233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:55.078428 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.078272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-tls\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:55.080610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.080581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d353d6cb-d949-4cdf-b425-0b1df26f10ea-node-exporter-tls\") pod \"node-exporter-jqgpc\" (UID: \"d353d6cb-d949-4cdf-b425-0b1df26f10ea\") " pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:55.080741 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.080615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f47eb749-c30c-4f90-aa6c-1bd247fca6b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-s4jt4\" (UID: \"f47eb749-c30c-4f90-aa6c-1bd247fca6b3\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:55.253660 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.253626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"
Sep 17 23:10:55.279905 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.279872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqgpc"
Sep 17 23:10:55.297817 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:55.297764    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd353d6cb_d949_4cdf_b425_0b1df26f10ea.slice/crio-fd473f9f2efe9c5381b70ea106392b7fc04327b1104f368d636e3843ef182b59 WatchSource:0}: Error finding container fd473f9f2efe9c5381b70ea106392b7fc04327b1104f368d636e3843ef182b59: Status 404 returned error can't find the container with id fd473f9f2efe9c5381b70ea106392b7fc04327b1104f368d636e3843ef182b59
Sep 17 23:10:55.533988 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.533832    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4"]
Sep 17 23:10:55.536513 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:10:55.536483    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47eb749_c30c_4f90_aa6c_1bd247fca6b3.slice/crio-0f229f77216f86c953f7a5978ad044d0755a2b01b7310aced01e3b6a361c8346 WatchSource:0}: Error finding container 0f229f77216f86c953f7a5978ad044d0755a2b01b7310aced01e3b6a361c8346: Status 404 returned error can't find the container with id 0f229f77216f86c953f7a5978ad044d0755a2b01b7310aced01e3b6a361c8346
Sep 17 23:10:55.590266 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.590239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqgpc" event={"ID":"d353d6cb-d949-4cdf-b425-0b1df26f10ea","Type":"ContainerStarted","Data":"fd473f9f2efe9c5381b70ea106392b7fc04327b1104f368d636e3843ef182b59"}
Sep 17 23:10:55.591936 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:55.591912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4" event={"ID":"f47eb749-c30c-4f90-aa6c-1bd247fca6b3","Type":"ContainerStarted","Data":"0f229f77216f86c953f7a5978ad044d0755a2b01b7310aced01e3b6a361c8346"}
Sep 17 23:10:56.541579 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.541514    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b4vqv"
Sep 17 23:10:56.597362 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.597326    2559 generic.go:356] "Generic (PLEG): container finished" podID="d353d6cb-d949-4cdf-b425-0b1df26f10ea" containerID="3e951688c62fe58c84706ac9e6bf92b387b093abef638a5afdf1236b8548ae74" exitCode=0
Sep 17 23:10:56.597841 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.597424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqgpc" event={"ID":"d353d6cb-d949-4cdf-b425-0b1df26f10ea","Type":"ContainerDied","Data":"3e951688c62fe58c84706ac9e6bf92b387b093abef638a5afdf1236b8548ae74"}
Sep 17 23:10:56.599660 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.599626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4" event={"ID":"f47eb749-c30c-4f90-aa6c-1bd247fca6b3","Type":"ContainerStarted","Data":"35fb5ac0a88630a1c89a707d782dcb600cca0fe0fc5a32267838fee10bb7e941"}
Sep 17 23:10:56.599769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.599684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4" event={"ID":"f47eb749-c30c-4f90-aa6c-1bd247fca6b3","Type":"ContainerStarted","Data":"bcc68881da8777fccdd659d61f443a98d4762c4da172f9ee273baf94a709f5c8"}
Sep 17 23:10:56.878309 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.878225    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:56.878309 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.878286    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:10:56.879684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.879657    2559 patch_prober.go:28] interesting pod/console-bd6cb97f7-nhwx8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.18:8443/health\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Sep 17 23:10:56.879814 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:56.879702    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-bd6cb97f7-nhwx8" podUID="21910f22-9119-4e58-a78f-ec5086cf2141" containerName="console" probeResult="failure" output="Get \"https://10.132.0.18:8443/health\": dial tcp 10.132.0.18:8443: connect: connection refused"
Sep 17 23:10:57.605205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:57.605167    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqgpc" event={"ID":"d353d6cb-d949-4cdf-b425-0b1df26f10ea","Type":"ContainerStarted","Data":"157884b1f23d69763540d78103d65bc9ca9f8a37be03918c21dc2cfe7605bce4"}
Sep 17 23:10:57.605664 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:57.605228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqgpc" event={"ID":"d353d6cb-d949-4cdf-b425-0b1df26f10ea","Type":"ContainerStarted","Data":"3cf9960f7dc3ec6a062fecdb97ba0b2a9d38813ac02700e7d2aec1a80f76d9c1"}
Sep 17 23:10:57.607155 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:57.607128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4" event={"ID":"f47eb749-c30c-4f90-aa6c-1bd247fca6b3","Type":"ContainerStarted","Data":"23bde96b60b1ffad8024aed885a5ff6ce1ae150bb06372de43cf3b6b956d83fe"}
Sep 17 23:10:57.625522 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:57.625479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jqgpc" podStartSLOduration=2.5765273669999997 podStartE2EDuration="3.625466409s" totalImagesPullingTime="980.201159ms" totalInitContainerRuntime="68.737883ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:57.623062612 +0000 UTC m=+300.097005595" watchObservedRunningTime="2026-09-17 23:10:57.625466409 +0000 UTC m=+300.099409419"
Sep 17 23:10:57.640670 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:57.640630    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-s4jt4" podStartSLOduration=2.711844033 podStartE2EDuration="3.640617435s" totalImagesPullingTime="928.773402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:57.639074695 +0000 UTC m=+300.113017678" watchObservedRunningTime="2026-09-17 23:10:57.640617435 +0000 UTC m=+300.114560417"
Sep 17 23:10:59.164512 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.164476    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd6cb97f7-nhwx8"]
Sep 17 23:10:59.529940 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.529908    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:10:59.530973 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.530954    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:10:59.537860 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.537840    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:10:59.538712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.538695    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:10:59.541542 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.541522    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:10:59.584313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.584202    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"]
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.588821    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.591310    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.591364    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.591378    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.591609    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.591621    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vrr45\""
Sep 17 23:10:59.592272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.591631    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 23:10:59.596449 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.596273    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 23:10:59.599783 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.599759    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"]
Sep 17 23:10:59.725834 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.725806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.725981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.725844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6v9\" (UniqueName: \"kubernetes.io/projected/c49c25c8-b81c-4e02-9195-7a3b7a93e689-kube-api-access-dn6v9\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.725981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.725869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-federate-client-tls\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.725981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.725936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-secret-telemeter-client\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.726102 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.725988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-metrics-client-ca\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.726102 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.726011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-telemeter-client-tls\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.726102 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.726063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-serving-certs-ca-bundle\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.726102 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.726081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827390 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-federate-client-tls\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827390 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-secret-telemeter-client\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827569 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-metrics-client-ca\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827569 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-telemeter-client-tls\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827569 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-serving-certs-ca-bundle\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827569 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827569 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.827853 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.827585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6v9\" (UniqueName: \"kubernetes.io/projected/c49c25c8-b81c-4e02-9195-7a3b7a93e689-kube-api-access-dn6v9\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.828325 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.828289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-serving-certs-ca-bundle\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.828447 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.828362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-metrics-client-ca\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.828447 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.828427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c25c8-b81c-4e02-9195-7a3b7a93e689-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.830068 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.830039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-federate-client-tls\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.830351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.830332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-secret-telemeter-client\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.830501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.830483    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.830539 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.830514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c49c25c8-b81c-4e02-9195-7a3b7a93e689-telemeter-client-tls\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.835326 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.835308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6v9\" (UniqueName: \"kubernetes.io/projected/c49c25c8-b81c-4e02-9195-7a3b7a93e689-kube-api-access-dn6v9\") pod \"telemeter-client-cbb4fb674-vxlkj\" (UID: \"c49c25c8-b81c-4e02-9195-7a3b7a93e689\") " pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:10:59.899165 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:10:59.899142    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"
Sep 17 23:11:00.182749 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.182703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cbb4fb674-vxlkj"]
Sep 17 23:11:00.184633 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:11:00.184606    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49c25c8_b81c_4e02_9195_7a3b7a93e689.slice/crio-55bb1208282c09f522d23b3910cb304c4a806508c5105b84d6a8503955e4b64e WatchSource:0}: Error finding container 55bb1208282c09f522d23b3910cb304c4a806508c5105b84d6a8503955e4b64e: Status 404 returned error can't find the container with id 55bb1208282c09f522d23b3910cb304c4a806508c5105b84d6a8503955e4b64e
Sep 17 23:11:00.319608 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.318375    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d84cd7677-wfvrz"]
Sep 17 23:11:00.326903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.326855    2559 patch_prober.go:28] interesting pod/image-registry-5d84cd7677-wfvrz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:11:00.327055 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.326935    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:11:00.533260 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.533227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:11:00.538427 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.538405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.541010 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.540974    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:11:00.541147 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.540972    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:11:00.541147 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lmpx9\""
Sep 17 23:11:00.541147 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.540979    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:11:00.541288 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541233    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:11:00.541450 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541429    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:11:00.541450 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541451    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:11:00.541787 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541761    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:11:00.541869 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541711    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.541968    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.542007    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.542020    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.542121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.542208    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a938q6a84ral4\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.542236    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:11:00.542344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.542306    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:11:00.543681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.543655    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:11:00.555383 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.555352    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:11:00.621067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.621032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj" event={"ID":"c49c25c8-b81c-4e02-9195-7a3b7a93e689","Type":"ContainerStarted","Data":"55bb1208282c09f522d23b3910cb304c4a806508c5105b84d6a8503955e4b64e"}
Sep 17 23:11:00.636792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.636758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.636938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.636837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrm9t\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-kube-api-access-lrm9t\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.636938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.636873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.636938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.636913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.636942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.636968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637195    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-config\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-config-out\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637692 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.637692 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.637435    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-web-config\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738582 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-config\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-config-out\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-web-config\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrm9t\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-kube-api-access-lrm9t\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.738777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.739126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.739126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.739126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.738851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.740410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.741367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.741002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.742819 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.742792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-config-out\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.743462 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.743443    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.743593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.743484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.745587 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.745499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.745796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.745775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-config\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.746512 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.746470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-web-config\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.746598 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.746520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.747163 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.747136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.747434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.747391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.747649 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.747627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.748161 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.748115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.748680 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.748641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.748863 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.748845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.749004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.748978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.753840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.752182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrm9t\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-kube-api-access-lrm9t\") pod \"prometheus-k8s-0\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:00.849606 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:00.849286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:01.143940 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:01.143866    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:11:01.146764 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:11:01.146736    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d857b5_3b08_417f_9704_d65714acfd30.slice/crio-3a4e161bb5fb3abcc4a0d454a8444e899d27d066472e5a36809fa4cdeb7c896d WatchSource:0}: Error finding container 3a4e161bb5fb3abcc4a0d454a8444e899d27d066472e5a36809fa4cdeb7c896d: Status 404 returned error can't find the container with id 3a4e161bb5fb3abcc4a0d454a8444e899d27d066472e5a36809fa4cdeb7c896d
Sep 17 23:11:01.149110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:01.149089    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:11:01.631570 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:01.631525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"3a4e161bb5fb3abcc4a0d454a8444e899d27d066472e5a36809fa4cdeb7c896d"}
Sep 17 23:11:02.637351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:02.637316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj" event={"ID":"c49c25c8-b81c-4e02-9195-7a3b7a93e689","Type":"ContainerStarted","Data":"7aa8879e73970ca42505c131e7983fde8e94ae999d00124d6b3313cc6f0f7718"}
Sep 17 23:11:02.834912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:02.834849    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:11:02.834912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:02.834899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:11:02.840615 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:02.840594    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:11:03.642517 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:03.642489    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56" exitCode=0
Sep 17 23:11:03.642991 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:03.642574    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56"}
Sep 17 23:11:03.644871 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:03.644842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj" event={"ID":"c49c25c8-b81c-4e02-9195-7a3b7a93e689","Type":"ContainerStarted","Data":"9c305e846fce0dac2070a75b286943eecfaf4833022b64024f02088d84db659d"}
Sep 17 23:11:03.645000 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:03.644896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj" event={"ID":"c49c25c8-b81c-4e02-9195-7a3b7a93e689","Type":"ContainerStarted","Data":"82d76cb710294a1a2332612c1f79c14a4ad329683bfe11244b99ef7599333fa5"}
Sep 17 23:11:03.648810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:03.648790    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:11:03.709714 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:03.709671    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cbb4fb674-vxlkj" podStartSLOduration=2.3779398130000002 podStartE2EDuration="4.709655291s" totalImagesPullingTime="2.331715478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:03.707567475 +0000 UTC m=+306.181510457" watchObservedRunningTime="2026-09-17 23:11:03.709655291 +0000 UTC m=+306.183598273"
Sep 17 23:11:05.633904 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:05.633867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:11:07.666462 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:07.666421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb"}
Sep 17 23:11:07.666882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:07.666513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379"}
Sep 17 23:11:09.676761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:09.676711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c"}
Sep 17 23:11:09.677229 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:09.676800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf"}
Sep 17 23:11:09.677229 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:09.676842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535"}
Sep 17 23:11:09.677229 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:09.676887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerStarted","Data":"9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50"}
Sep 17 23:11:09.705446 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:09.705392    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.859028086 podStartE2EDuration="9.70537497s" totalImagesPullingTime="6.823252328s" totalInitContainerRuntime="23.094556ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:09.703254776 +0000 UTC m=+312.177197758" watchObservedRunningTime="2026-09-17 23:11:09.70537497 +0000 UTC m=+312.179317954"
Sep 17 23:11:10.323164 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:10.323132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:11:10.850167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:10.850130    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:11:14.980538 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:14.980504    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b8c7db9df-dz6pz"]
Sep 17 23:11:24.184693 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.184656    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bd6cb97f7-nhwx8" podUID="21910f22-9119-4e58-a78f-ec5086cf2141" containerName="console" containerID="cri-o://7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48" gracePeriod=15
Sep 17 23:11:24.420835 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.420810    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:11:24.561031 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-service-ca\") pod \"21910f22-9119-4e58-a78f-ec5086cf2141\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") "
Sep 17 23:11:24.561145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-oauth-serving-cert\") pod \"21910f22-9119-4e58-a78f-ec5086cf2141\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") "
Sep 17 23:11:24.561145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561064    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-console-config\") pod \"21910f22-9119-4e58-a78f-ec5086cf2141\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") "
Sep 17 23:11:24.561145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdrvm\" (UniqueName: \"kubernetes.io/projected/21910f22-9119-4e58-a78f-ec5086cf2141-kube-api-access-wdrvm\") pod \"21910f22-9119-4e58-a78f-ec5086cf2141\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") "
Sep 17 23:11:24.561299 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561158    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-serving-cert\") pod \"21910f22-9119-4e58-a78f-ec5086cf2141\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") "
Sep 17 23:11:24.561299 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561231    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-oauth-config\") pod \"21910f22-9119-4e58-a78f-ec5086cf2141\" (UID: \"21910f22-9119-4e58-a78f-ec5086cf2141\") "
Sep 17 23:11:24.561462 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561434    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "21910f22-9119-4e58-a78f-ec5086cf2141" (UID: "21910f22-9119-4e58-a78f-ec5086cf2141"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:24.561529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561441    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-service-ca" (OuterVolumeSpecName: "service-ca") pod "21910f22-9119-4e58-a78f-ec5086cf2141" (UID: "21910f22-9119-4e58-a78f-ec5086cf2141"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:24.561529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.561486    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-console-config" (OuterVolumeSpecName: "console-config") pod "21910f22-9119-4e58-a78f-ec5086cf2141" (UID: "21910f22-9119-4e58-a78f-ec5086cf2141"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:24.563436 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.563404    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "21910f22-9119-4e58-a78f-ec5086cf2141" (UID: "21910f22-9119-4e58-a78f-ec5086cf2141"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:11:24.563557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.563444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "21910f22-9119-4e58-a78f-ec5086cf2141" (UID: "21910f22-9119-4e58-a78f-ec5086cf2141"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:11:24.563557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.563468    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21910f22-9119-4e58-a78f-ec5086cf2141-kube-api-access-wdrvm" (OuterVolumeSpecName: "kube-api-access-wdrvm") pod "21910f22-9119-4e58-a78f-ec5086cf2141" (UID: "21910f22-9119-4e58-a78f-ec5086cf2141"). InnerVolumeSpecName "kube-api-access-wdrvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:11:24.662764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.662737    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-service-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:24.662764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.662762    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-oauth-serving-cert\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:24.662909 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.662777    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21910f22-9119-4e58-a78f-ec5086cf2141-console-config\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:24.662909 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.662790    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrvm\" (UniqueName: \"kubernetes.io/projected/21910f22-9119-4e58-a78f-ec5086cf2141-kube-api-access-wdrvm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:24.662909 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.662801    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-serving-cert\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:24.662909 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.662813    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21910f22-9119-4e58-a78f-ec5086cf2141-console-oauth-config\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:24.735871 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.735849    2559 generic.go:356] "Generic (PLEG): container finished" podID="21910f22-9119-4e58-a78f-ec5086cf2141" containerID="7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48" exitCode=0
Sep 17 23:11:24.735959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.735893    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd6cb97f7-nhwx8"
Sep 17 23:11:24.735959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.735907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd6cb97f7-nhwx8" event={"ID":"21910f22-9119-4e58-a78f-ec5086cf2141","Type":"ContainerDied","Data":"7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48"}
Sep 17 23:11:24.735959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.735944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd6cb97f7-nhwx8" event={"ID":"21910f22-9119-4e58-a78f-ec5086cf2141","Type":"ContainerDied","Data":"5ad78b8c0a5c4c6804566a60c45ccd6774be1cc83cf4b314f190250a34642cca"}
Sep 17 23:11:24.736059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.735963    2559 scope.go:117] "RemoveContainer" containerID="7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48"
Sep 17 23:11:24.744524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.744507    2559 scope.go:117] "RemoveContainer" containerID="7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48"
Sep 17 23:11:24.744805 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:11:24.744785    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48\": container with ID starting with 7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48 not found: ID does not exist" containerID="7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48"
Sep 17 23:11:24.744857 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.744812    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48"} err="failed to get container status \"7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48\": rpc error: code = NotFound desc = could not find container \"7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48\": container with ID starting with 7641243e77a1787175a1d95e1c5101bd34ef3983ccf0505a676f213905cddf48 not found: ID does not exist"
Sep 17 23:11:24.756859 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.756840    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd6cb97f7-nhwx8"]
Sep 17 23:11:24.760626 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:24.760605    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bd6cb97f7-nhwx8"]
Sep 17 23:11:25.340175 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.340140    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" containerName="registry" containerID="cri-o://4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136" gracePeriod=30
Sep 17 23:11:25.601579 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.601531    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:11:25.638116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.638084    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21910f22-9119-4e58-a78f-ec5086cf2141" path="/var/lib/kubelet/pods/21910f22-9119-4e58-a78f-ec5086cf2141/volumes"
Sep 17 23:11:25.679033 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679010    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-certificates\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679157 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-trusted-ca\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679157 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679091    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-ca-trust-extracted\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-installation-pull-secrets\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679267    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-image-registry-private-configuration\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679323    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpc5d\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-kube-api-access-hpc5d\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679354    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-bound-sa-token\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679492 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679398    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") pod \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\" (UID: \"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a\") "
Sep 17 23:11:25.679492 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679451    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:25.679649 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679540    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:25.679702 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679676    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-certificates\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.679702 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.679695    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-trusted-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.681796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.681763    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:11:25.681796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.681775    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:11:25.681957 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.681841    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:11:25.682011 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.681975    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:11:25.682266 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.682243    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-kube-api-access-hpc5d" (OuterVolumeSpecName: "kube-api-access-hpc5d") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "kube-api-access-hpc5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:11:25.687635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.687608    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" (UID: "86ddb0e4-cd95-4fde-be49-09bffe7d9d1a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:11:25.741064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.741043    2559 generic.go:356] "Generic (PLEG): container finished" podID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" containerID="4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136" exitCode=0
Sep 17 23:11:25.741151 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.741083    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz"
Sep 17 23:11:25.741151 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.741134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" event={"ID":"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a","Type":"ContainerDied","Data":"4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136"}
Sep 17 23:11:25.741248 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.741175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d84cd7677-wfvrz" event={"ID":"86ddb0e4-cd95-4fde-be49-09bffe7d9d1a","Type":"ContainerDied","Data":"1cdbce587e52aabbf85547e42f0a5a368733a00c410ec8ca68c3d6f993eb0e20"}
Sep 17 23:11:25.741248 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.741196    2559 scope.go:117] "RemoveContainer" containerID="4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136"
Sep 17 23:11:25.749095 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.749080    2559 scope.go:117] "RemoveContainer" containerID="4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136"
Sep 17 23:11:25.749370 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:11:25.749351    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136\": container with ID starting with 4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136 not found: ID does not exist" containerID="4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136"
Sep 17 23:11:25.749446 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.749377    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136"} err="failed to get container status \"4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136\": rpc error: code = NotFound desc = could not find container \"4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136\": container with ID starting with 4e39a9f867a3d0d577de122ad5db399d5332ee89bb3ad37a4ea82601676a8136 not found: ID does not exist"
Sep 17 23:11:25.761308 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.761288    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d84cd7677-wfvrz"]
Sep 17 23:11:25.766096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.766075    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d84cd7677-wfvrz"]
Sep 17 23:11:25.780065 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.780047    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-installation-pull-secrets\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.780131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.780071    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-image-registry-private-configuration\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.780131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.780086    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpc5d\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-kube-api-access-hpc5d\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.780131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.780098    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-bound-sa-token\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.780131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.780111    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-registry-tls\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:25.780131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:25.780122    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a-ca-trust-extracted\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:27.637929 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:27.637896    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" path="/var/lib/kubelet/pods/86ddb0e4-cd95-4fde-be49-09bffe7d9d1a/volumes"
Sep 17 23:11:40.002287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.002234    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b8c7db9df-dz6pz" podUID="68f92918-bba6-4d9f-ae33-554b3d51652d" containerName="console" containerID="cri-o://073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8" gracePeriod=15
Sep 17 23:11:40.247969 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.247945    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:11:40.380280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380205    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfhvj\" (UniqueName: \"kubernetes.io/projected/68f92918-bba6-4d9f-ae33-554b3d51652d-kube-api-access-gfhvj\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-trusted-ca-bundle\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380282    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-oauth-serving-cert\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380335    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-serving-cert\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380372    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-service-ca\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-oauth-config\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-console-config\") pod \"68f92918-bba6-4d9f-ae33-554b3d51652d\" (UID: \"68f92918-bba6-4d9f-ae33-554b3d51652d\") "
Sep 17 23:11:40.380789 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:40.380789 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380773    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:40.381027 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.380995    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-service-ca" (OuterVolumeSpecName: "service-ca") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:40.381126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.381006    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-console-config" (OuterVolumeSpecName: "console-config") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:11:40.382414 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.382385    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f92918-bba6-4d9f-ae33-554b3d51652d-kube-api-access-gfhvj" (OuterVolumeSpecName: "kube-api-access-gfhvj") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "kube-api-access-gfhvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:11:40.382506 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.382485    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:11:40.382587 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.382569    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "68f92918-bba6-4d9f-ae33-554b3d51652d" (UID: "68f92918-bba6-4d9f-ae33-554b3d51652d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:11:40.481433 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481414    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-oauth-config\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.481433 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481433    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-console-config\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.481557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481443    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhvj\" (UniqueName: \"kubernetes.io/projected/68f92918-bba6-4d9f-ae33-554b3d51652d-kube-api-access-gfhvj\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.481557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481452    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-trusted-ca-bundle\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.481557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481460    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-oauth-serving-cert\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.481557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481469    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68f92918-bba6-4d9f-ae33-554b3d51652d-console-serving-cert\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.481557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.481477    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68f92918-bba6-4d9f-ae33-554b3d51652d-service-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:11:40.812785 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.812756    2559 generic.go:356] "Generic (PLEG): container finished" podID="68f92918-bba6-4d9f-ae33-554b3d51652d" containerID="073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8" exitCode=0
Sep 17 23:11:40.812924 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.812797    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c7db9df-dz6pz"
Sep 17 23:11:40.812924 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.812845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c7db9df-dz6pz" event={"ID":"68f92918-bba6-4d9f-ae33-554b3d51652d","Type":"ContainerDied","Data":"073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8"}
Sep 17 23:11:40.812924 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.812896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c7db9df-dz6pz" event={"ID":"68f92918-bba6-4d9f-ae33-554b3d51652d","Type":"ContainerDied","Data":"4718a07be187085aa5d0539555879e789d24f206243312cfaf8c8e68cd9cf3c6"}
Sep 17 23:11:40.812924 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.812920    2559 scope.go:117] "RemoveContainer" containerID="073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8"
Sep 17 23:11:40.825804 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.825786    2559 scope.go:117] "RemoveContainer" containerID="073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8"
Sep 17 23:11:40.826083 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:11:40.826049    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8\": container with ID starting with 073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8 not found: ID does not exist" containerID="073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8"
Sep 17 23:11:40.826176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.826088    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8"} err="failed to get container status \"073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8\": rpc error: code = NotFound desc = could not find container \"073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8\": container with ID starting with 073b61aa2f9479d636962e045b2389690222dfe9809d2acc32c73d050546e9f8 not found: ID does not exist"
Sep 17 23:11:40.835663 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.835641    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b8c7db9df-dz6pz"]
Sep 17 23:11:40.839327 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:40.839307    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b8c7db9df-dz6pz"]
Sep 17 23:11:41.638513 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:41.638488    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f92918-bba6-4d9f-ae33-554b3d51652d" path="/var/lib/kubelet/pods/68f92918-bba6-4d9f-ae33-554b3d51652d/volumes"
Sep 17 23:11:43.827157 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:43.827128    2559 generic.go:356] "Generic (PLEG): container finished" podID="fe98b390-a870-4ae3-9656-cb7d1dc56c29" containerID="229df4b7524bfde22aad1b1a8b84b0f9a3b9c4443769cf62d35d7d87c1f4c7dd" exitCode=0
Sep 17 23:11:43.827564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:43.827205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54" event={"ID":"fe98b390-a870-4ae3-9656-cb7d1dc56c29","Type":"ContainerDied","Data":"229df4b7524bfde22aad1b1a8b84b0f9a3b9c4443769cf62d35d7d87c1f4c7dd"}
Sep 17 23:11:43.827627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:43.827600    2559 scope.go:117] "RemoveContainer" containerID="229df4b7524bfde22aad1b1a8b84b0f9a3b9c4443769cf62d35d7d87c1f4c7dd"
Sep 17 23:11:44.832645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:44.832565    2559 generic.go:356] "Generic (PLEG): container finished" podID="7fdf7271-1621-4a9f-8b5c-78fed07d772f" containerID="4b28b62a95a3ec7e29995f78e6e7186dedae7c68d4d652d34f263db8c62df909" exitCode=0
Sep 17 23:11:44.833046 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:44.832641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9" event={"ID":"7fdf7271-1621-4a9f-8b5c-78fed07d772f","Type":"ContainerDied","Data":"4b28b62a95a3ec7e29995f78e6e7186dedae7c68d4d652d34f263db8c62df909"}
Sep 17 23:11:44.833046 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:44.832996    2559 scope.go:117] "RemoveContainer" containerID="4b28b62a95a3ec7e29995f78e6e7186dedae7c68d4d652d34f263db8c62df909"
Sep 17 23:11:44.834757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:44.834708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x5n54" event={"ID":"fe98b390-a870-4ae3-9656-cb7d1dc56c29","Type":"ContainerStarted","Data":"0768a626164e9ece2af0b1278df141237af63f169822313803f2686eef75f376"}
Sep 17 23:11:45.840347 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:11:45.840311    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2v7x9" event={"ID":"7fdf7271-1621-4a9f-8b5c-78fed07d772f","Type":"ContainerStarted","Data":"77c2b18b101f333a710d3ad5a0ba6d6f53513085254279cd1b51928528b89681"}
Sep 17 23:12:00.849673 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:00.849639    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:00.865527 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:00.865505    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:00.919602 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:00.919549    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:12.526343 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:12.526302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:12:12.529006 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:12.528975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4404befb-c49e-440b-9f60-9b04e47390d0-metrics-certs\") pod \"network-metrics-daemon-kd746\" (UID: \"4404befb-c49e-440b-9f60-9b04e47390d0\") " pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:12:12.537055 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:12.537028    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hrj9x\""
Sep 17 23:12:12.545335 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:12.545310    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kd746"
Sep 17 23:12:12.817062 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:12.816998    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kd746"]
Sep 17 23:12:12.820574 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:12:12.820550    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4404befb_c49e_440b_9f60_9b04e47390d0.slice/crio-fe6d459d4220feb67f9093c00f2926be59c5434031f36203326a5b8aec739f5a WatchSource:0}: Error finding container fe6d459d4220feb67f9093c00f2926be59c5434031f36203326a5b8aec739f5a: Status 404 returned error can't find the container with id fe6d459d4220feb67f9093c00f2926be59c5434031f36203326a5b8aec739f5a
Sep 17 23:12:12.950625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:12.950596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kd746" event={"ID":"4404befb-c49e-440b-9f60-9b04e47390d0","Type":"ContainerStarted","Data":"fe6d459d4220feb67f9093c00f2926be59c5434031f36203326a5b8aec739f5a"}
Sep 17 23:12:14.966338 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:14.966291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kd746" event={"ID":"4404befb-c49e-440b-9f60-9b04e47390d0","Type":"ContainerStarted","Data":"6c97d3ae474b654595fd1404d1e59465a59d9ecebb340dcda1d0d7e5f543ff01"}
Sep 17 23:12:14.966769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:14.966365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kd746" event={"ID":"4404befb-c49e-440b-9f60-9b04e47390d0","Type":"ContainerStarted","Data":"9151ab1fc944042eeba80304186ec13a60e0f7c076f1d9e96b92db83de18cd93"}
Sep 17 23:12:14.982783 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:14.982714    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kd746" podStartSLOduration=374.538682645 podStartE2EDuration="6m15.98270034s" totalImagesPullingTime="1.444017695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:14.98097713 +0000 UTC m=+377.454920112" watchObservedRunningTime="2026-09-17 23:12:14.98270034 +0000 UTC m=+377.456643324"
Sep 17 23:12:18.929662 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.929621    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:18.930110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.929971    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="prometheus" containerID="cri-o://bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379" gracePeriod=600
Sep 17 23:12:18.930110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.930021    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="thanos-sidecar" containerID="cri-o://9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50" gracePeriod=600
Sep 17 23:12:18.930110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.930068    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-web" containerID="cri-o://bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535" gracePeriod=600
Sep 17 23:12:18.930306 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.930089    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="config-reloader" containerID="cri-o://65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb" gracePeriod=600
Sep 17 23:12:18.930306 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.930059    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy" containerID="cri-o://486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf" gracePeriod=600
Sep 17 23:12:18.930306 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:18.930146    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c" gracePeriod=600
Sep 17 23:12:19.991593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991562    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c" exitCode=0
Sep 17 23:12:19.991593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991587    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf" exitCode=0
Sep 17 23:12:19.991593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991593    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50" exitCode=0
Sep 17 23:12:19.991593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991602    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb" exitCode=0
Sep 17 23:12:19.991593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991607    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379" exitCode=0
Sep 17 23:12:19.992088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c"}
Sep 17 23:12:19.992088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf"}
Sep 17 23:12:19.992088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50"}
Sep 17 23:12:19.992088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb"}
Sep 17 23:12:19.992088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:19.991805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379"}
Sep 17 23:12:20.178159 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.178133    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:20.290334 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290259    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290334 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290292    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-trusted-ca-bundle\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290334 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290326    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-tls\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290342    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-1\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290357    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-metrics-client-certs\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290428    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-0\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-kube-rbac-proxy\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-config-out\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290541    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-config\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-grpc-tls\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290630    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-kubelet-serving-ca-bundle\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290657    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-db\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290703    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-serving-certs-ca-bundle\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290744    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-thanos-prometheus-http-client-file\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290778    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290793    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-metrics-client-ca\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290799    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-web-config\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290849    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-tls-assets\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290920    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrm9t\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-kube-api-access-lrm9t\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.290955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.290948    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-2\") pod \"62d857b5-3b08-417f-9704-d65714acfd30\" (UID: \"62d857b5-3b08-417f-9704-d65714acfd30\") "
Sep 17 23:12:20.291521 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.291249    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.291521 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.291267    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.291616 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.291602    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.292198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.291945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.292198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.292008    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:12:20.292198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.292154    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.292819 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.292767    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.293070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.293026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:20.293828 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.293791    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.294152 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.294129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.294679 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.294641    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.295337 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.295274    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.295337 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.295305    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.295452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.295352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-config" (OuterVolumeSpecName: "config") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.295668 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.295645    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:12:20.295971 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.295941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.296043 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.295965    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.296082 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.296045    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-kube-api-access-lrm9t" (OuterVolumeSpecName: "kube-api-access-lrm9t") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "kube-api-access-lrm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:12:20.296623 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.296603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-config-out" (OuterVolumeSpecName: "config-out") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:12:20.305836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.305817    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-web-config" (OuterVolumeSpecName: "web-config") pod "62d857b5-3b08-417f-9704-d65714acfd30" (UID: "62d857b5-3b08-417f-9704-d65714acfd30"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:20.391798 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391775    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrm9t\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-kube-api-access-lrm9t\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.391798 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391794    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.391942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391804    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.391942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391813    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.391942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391821    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-metrics-client-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.391942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391830    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.391958    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-kube-rbac-proxy\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392203    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-config-out\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392218    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-config\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392230    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-grpc-tls\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392253    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392267    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62d857b5-3b08-417f-9704-d65714acfd30-prometheus-k8s-db\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392281    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392334    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392353    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62d857b5-3b08-417f-9704-d65714acfd30-configmap-metrics-client-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392369    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-web-config\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392390    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62d857b5-3b08-417f-9704-d65714acfd30-tls-assets\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.397195 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.392405    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62d857b5-3b08-417f-9704-d65714acfd30-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:12:20.999838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.999802    2559 generic.go:356] "Generic (PLEG): container finished" podID="62d857b5-3b08-417f-9704-d65714acfd30" containerID="bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535" exitCode=0
Sep 17 23:12:21.000286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.999860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535"}
Sep 17 23:12:21.000286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.999908    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.000286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:20.999944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62d857b5-3b08-417f-9704-d65714acfd30","Type":"ContainerDied","Data":"3a4e161bb5fb3abcc4a0d454a8444e899d27d066472e5a36809fa4cdeb7c896d"}
Sep 17 23:12:21.000286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.000002    2559 scope.go:117] "RemoveContainer" containerID="c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c"
Sep 17 23:12:21.009588 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.009560    2559 scope.go:117] "RemoveContainer" containerID="486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf"
Sep 17 23:12:21.021780 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.021753    2559 scope.go:117] "RemoveContainer" containerID="bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535"
Sep 17 23:12:21.026850 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.026816    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:21.030235 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.030053    2559 scope.go:117] "RemoveContainer" containerID="9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50"
Sep 17 23:12:21.032251 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.032229    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:21.037779 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.037762    2559 scope.go:117] "RemoveContainer" containerID="65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb"
Sep 17 23:12:21.044489 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.044472    2559 scope.go:117] "RemoveContainer" containerID="bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379"
Sep 17 23:12:21.052101 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.051943    2559 scope.go:117] "RemoveContainer" containerID="2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56"
Sep 17 23:12:21.058942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.058925    2559 scope.go:117] "RemoveContainer" containerID="c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c"
Sep 17 23:12:21.059220 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.059200    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c\": container with ID starting with c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c not found: ID does not exist" containerID="c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c"
Sep 17 23:12:21.059300 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059233    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c"} err="failed to get container status \"c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c\": rpc error: code = NotFound desc = could not find container \"c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c\": container with ID starting with c1b4acc50fc53349c566b64482f34f534603d28fbfb6d04ff97469111cdb867c not found: ID does not exist"
Sep 17 23:12:21.059300 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059258    2559 scope.go:117] "RemoveContainer" containerID="486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf"
Sep 17 23:12:21.059588 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.059571    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf\": container with ID starting with 486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf not found: ID does not exist" containerID="486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf"
Sep 17 23:12:21.059629 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059597    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf"} err="failed to get container status \"486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf\": rpc error: code = NotFound desc = could not find container \"486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf\": container with ID starting with 486012ae47ba304c96a6944859b327c7c68c1c627d0ba4915a765a2c2ffadcbf not found: ID does not exist"
Sep 17 23:12:21.059629 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059614    2559 scope.go:117] "RemoveContainer" containerID="bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535"
Sep 17 23:12:21.059867 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.059848    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535\": container with ID starting with bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535 not found: ID does not exist" containerID="bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535"
Sep 17 23:12:21.059951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059875    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535"} err="failed to get container status \"bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535\": rpc error: code = NotFound desc = could not find container \"bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535\": container with ID starting with bce2dd8e65a29ae79adbb7f9074fa908b09ef3fb590602b23b0dd620a4f22535 not found: ID does not exist"
Sep 17 23:12:21.059951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059887    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:21.059951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.059898    2559 scope.go:117] "RemoveContainer" containerID="9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50"
Sep 17 23:12:21.060132 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.060116    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50\": container with ID starting with 9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50 not found: ID does not exist" containerID="9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50"
Sep 17 23:12:21.060193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060137    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50"} err="failed to get container status \"9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50\": rpc error: code = NotFound desc = could not find container \"9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50\": container with ID starting with 9d7965be2ea9cc482afa05df442aff37214549c39721f0e2f1624072aade1c50 not found: ID does not exist"
Sep 17 23:12:21.060193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060156    2559 scope.go:117] "RemoveContainer" containerID="65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060213    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-web"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060226    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-web"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060235    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-thanos"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060241    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-thanos"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060249    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21910f22-9119-4e58-a78f-ec5086cf2141" containerName="console"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060254    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21910f22-9119-4e58-a78f-ec5086cf2141" containerName="console"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060261    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f92918-bba6-4d9f-ae33-554b3d51652d" containerName="console"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060266    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f92918-bba6-4d9f-ae33-554b3d51652d" containerName="console"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060273    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="thanos-sidecar"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060280    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="thanos-sidecar"
Sep 17 23:12:21.060287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060292    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="init-config-reloader"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060300    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="init-config-reloader"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060314    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060322    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060335    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" containerName="registry"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060341    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" containerName="registry"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060350    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="prometheus"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="prometheus"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060364    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="config-reloader"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060369    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="config-reloader"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.060379    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb\": container with ID starting with 65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb not found: ID does not exist" containerID="65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060395    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb"} err="failed to get container status \"65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb\": rpc error: code = NotFound desc = could not find container \"65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb\": container with ID starting with 65f9325a47677a35e6546c86b8daa035b34e4acfb474fa381439a5192a8f68cb not found: ID does not exist"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060408    2559 scope.go:117] "RemoveContainer" containerID="bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060419    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f92918-bba6-4d9f-ae33-554b3d51652d" containerName="console"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060426    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="21910f22-9119-4e58-a78f-ec5086cf2141" containerName="console"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060432    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-thanos"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060438    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="config-reloader"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060444    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060450    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="prometheus"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060455    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ddb0e4-cd95-4fde-be49-09bffe7d9d1a" containerName="registry"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060460    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="thanos-sidecar"
Sep 17 23:12:21.060646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060465    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d857b5-3b08-417f-9704-d65714acfd30" containerName="kube-rbac-proxy-web"
Sep 17 23:12:21.061352 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.060683    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379\": container with ID starting with bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379 not found: ID does not exist" containerID="bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379"
Sep 17 23:12:21.061352 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060703    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379"} err="failed to get container status \"bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379\": rpc error: code = NotFound desc = could not find container \"bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379\": container with ID starting with bbc731d5839ffee2b9be2c27d04f8d5c5bc45a8cadee48250b9815a81b24c379 not found: ID does not exist"
Sep 17 23:12:21.061352 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.060732    2559 scope.go:117] "RemoveContainer" containerID="2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56"
Sep 17 23:12:21.061352 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:12:21.060982    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56\": container with ID starting with 2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56 not found: ID does not exist" containerID="2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56"
Sep 17 23:12:21.061352 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.061009    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56"} err="failed to get container status \"2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56\": rpc error: code = NotFound desc = could not find container \"2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56\": container with ID starting with 2438cf4f3728381c63559ece59ac220b4ffe473d0335fcc03d9917eff415df56 not found: ID does not exist"
Sep 17 23:12:21.065375 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.065356    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.067975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.067951    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:12:21.067975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.067971    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:12:21.068142 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068094    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:12:21.068142 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068131    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a938q6a84ral4\""
Sep 17 23:12:21.068398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068350    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:12:21.068398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:12:21.068398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068375    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:12:21.068398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068398    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:12:21.068672 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068474    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lmpx9\""
Sep 17 23:12:21.068777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:12:21.068958 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.068939    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:12:21.069058 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.069039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:12:21.069121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.069062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:12:21.069121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.069068    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:12:21.069252 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.069155    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:12:21.072264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.072244    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:12:21.073940 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.073921    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:12:21.083969 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.083940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:21.197601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hht\" (UniqueName: \"kubernetes.io/projected/5db953bb-c30d-42f2-a534-d048ee960b12-kube-api-access-h8hht\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197949 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197949 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.197949 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198034 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198034 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.197976    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198034 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-web-config\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5db953bb-c30d-42f2-a534-d048ee960b12-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-config\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5db953bb-c30d-42f2-a534-d048ee960b12-config-out\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.198207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.198185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.298881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.298827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.298881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.298856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.298881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.298874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299031 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.298895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5db953bb-c30d-42f2-a534-d048ee960b12-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299031 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.298921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-config\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299107 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5db953bb-c30d-42f2-a534-d048ee960b12-config-out\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299107 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299206 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299206 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299206 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299206 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hht\" (UniqueName: \"kubernetes.io/projected/5db953bb-c30d-42f2-a534-d048ee960b12-kube-api-access-h8hht\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299252    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299647 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299647 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299647 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299647 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-web-config\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.299647 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.299599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.301387 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.300407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.301387 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.301063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.302182 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.302148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.302182 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.302172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-config\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.302349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.302307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5db953bb-c30d-42f2-a534-d048ee960b12-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.302349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.302318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.302458 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.302434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5db953bb-c30d-42f2-a534-d048ee960b12-config-out\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.302521 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.302462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.303062 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.303032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.303158 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.303084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.303222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.303157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.303222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.303159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-web-config\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.303803 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.303776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.304110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.304084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.305200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.305176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.305794 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.305771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.305870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.305820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5db953bb-c30d-42f2-a534-d048ee960b12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.306171 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.306150    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5db953bb-c30d-42f2-a534-d048ee960b12-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.311741 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.311708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hht\" (UniqueName: \"kubernetes.io/projected/5db953bb-c30d-42f2-a534-d048ee960b12-kube-api-access-h8hht\") pod \"prometheus-k8s-0\" (UID: \"5db953bb-c30d-42f2-a534-d048ee960b12\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.375362 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.375339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:21.638733 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.638654    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d857b5-3b08-417f-9704-d65714acfd30" path="/var/lib/kubelet/pods/62d857b5-3b08-417f-9704-d65714acfd30/volumes"
Sep 17 23:12:21.661420 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:21.661392    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:21.664953 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:12:21.664926    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db953bb_c30d_42f2_a534_d048ee960b12.slice/crio-695681d0697db0d26c4869392888aedd25d10b8d267acc2d3b8936244a640e16 WatchSource:0}: Error finding container 695681d0697db0d26c4869392888aedd25d10b8d267acc2d3b8936244a640e16: Status 404 returned error can't find the container with id 695681d0697db0d26c4869392888aedd25d10b8d267acc2d3b8936244a640e16
Sep 17 23:12:22.006704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:22.006663    2559 generic.go:356] "Generic (PLEG): container finished" podID="5db953bb-c30d-42f2-a534-d048ee960b12" containerID="1712a61d0799aad8121c64b07ea96a382b7b1e7323836d7370e6d6e1cb387358" exitCode=0
Sep 17 23:12:22.007209 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:22.006752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerDied","Data":"1712a61d0799aad8121c64b07ea96a382b7b1e7323836d7370e6d6e1cb387358"}
Sep 17 23:12:22.007209 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:22.006851    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"695681d0697db0d26c4869392888aedd25d10b8d267acc2d3b8936244a640e16"}
Sep 17 23:12:23.016548 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.016502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"03c3cbe0189d0240eef09946e89970f344e0e365e6ebb6e48f74c6044c57d471"}
Sep 17 23:12:23.016956 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.016599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"317b7bbd73a7c8f5408233ee59dc49a0ddc10583fe5ac13afc3b410e94a96bae"}
Sep 17 23:12:23.016956 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.016643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"a53aaa39654e1b1d36432490fc6302fb9cebec2ead2ff113b7a8809fb60c67cc"}
Sep 17 23:12:23.016956 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.016683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"a55755faa55fe42d06654bc9968213f02fdb1ed43f57ec244d7e54d065d12a89"}
Sep 17 23:12:23.016956 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.016749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"27524e75610fde3f706fcd7386812efda0dabfdf1b016c4b6596f52b16953ebc"}
Sep 17 23:12:23.016956 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.016789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db953bb-c30d-42f2-a534-d048ee960b12","Type":"ContainerStarted","Data":"ee772d087aaf8179219afd16729577b97d652faad70a102a968dc77eef9ecb2f"}
Sep 17 23:12:23.044426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:23.044367    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.020196618 podStartE2EDuration="2.044345551s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.148933ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:23.042981195 +0000 UTC m=+385.516924179" watchObservedRunningTime="2026-09-17 23:12:23.044345551 +0000 UTC m=+385.518288535"
Sep 17 23:12:26.376127 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:12:26.376086    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:21.375704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:13:21.375674    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:21.390955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:13:21.390927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:22.282491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:13:22.282452    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:06.279778 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.279745    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"]
Sep 17 23:14:06.283275 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.283258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.285313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.285287    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 23:14:06.285429 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.285342    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:06.285480 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.285450    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xcrc6\""
Sep 17 23:14:06.290137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.290116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"]
Sep 17 23:14:06.331943 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.331919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6jk\" (UniqueName: \"kubernetes.io/projected/9edc9f7f-6817-45fd-8de5-ddff0ae1b59f-kube-api-access-5r6jk\") pod \"openshift-lws-operator-fd8ccff4c-6hg4n\" (UID: \"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.332028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.331950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9edc9f7f-6817-45fd-8de5-ddff0ae1b59f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6hg4n\" (UID: \"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.432992 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.432971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6jk\" (UniqueName: \"kubernetes.io/projected/9edc9f7f-6817-45fd-8de5-ddff0ae1b59f-kube-api-access-5r6jk\") pod \"openshift-lws-operator-fd8ccff4c-6hg4n\" (UID: \"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.433083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.433001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9edc9f7f-6817-45fd-8de5-ddff0ae1b59f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6hg4n\" (UID: \"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.433267 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.433254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9edc9f7f-6817-45fd-8de5-ddff0ae1b59f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6hg4n\" (UID: \"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.441740 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.441703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6jk\" (UniqueName: \"kubernetes.io/projected/9edc9f7f-6817-45fd-8de5-ddff0ae1b59f-kube-api-access-5r6jk\") pod \"openshift-lws-operator-fd8ccff4c-6hg4n\" (UID: \"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.603182 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.603135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"
Sep 17 23:14:06.871825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:06.871799    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n"]
Sep 17 23:14:06.874071 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:14:06.874047    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9edc9f7f_6817_45fd_8de5_ddff0ae1b59f.slice/crio-9892ee6204565a6638c8a2758d5b0c85d9a74fcfd4641162cfa0bbef12c1980d WatchSource:0}: Error finding container 9892ee6204565a6638c8a2758d5b0c85d9a74fcfd4641162cfa0bbef12c1980d: Status 404 returned error can't find the container with id 9892ee6204565a6638c8a2758d5b0c85d9a74fcfd4641162cfa0bbef12c1980d
Sep 17 23:14:07.438105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:07.438075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n" event={"ID":"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f","Type":"ContainerStarted","Data":"9892ee6204565a6638c8a2758d5b0c85d9a74fcfd4641162cfa0bbef12c1980d"}
Sep 17 23:14:09.447605 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:09.447572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n" event={"ID":"9edc9f7f-6817-45fd-8de5-ddff0ae1b59f","Type":"ContainerStarted","Data":"a9754b5c3dedcb728c22a1f0bbd2988714c83a36f2c5eb47fce7f2ac35f68d39"}
Sep 17 23:14:09.463433 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:09.463384    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6hg4n" podStartSLOduration=1.096311768 podStartE2EDuration="3.463370102s" totalImagesPullingTime="2.367058334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:09.461556825 +0000 UTC m=+491.935499807" watchObservedRunningTime="2026-09-17 23:14:09.463370102 +0000 UTC m=+491.937313085"
Sep 17 23:14:27.257072 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.257044    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"]
Sep 17 23:14:27.263057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.263040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.265262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.265241    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 23:14:27.265665 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.265645    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 23:14:27.265790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.265737    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-24qcm\""
Sep 17 23:14:27.265926 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.265913    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 23:14:27.265984 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.265967    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 23:14:27.280111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.280090    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"]
Sep 17 23:14:27.383617 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.383586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ddca1d2-1117-442d-a537-26515c0cacab-tmp\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.383769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.383633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ddca1d2-1117-442d-a537-26515c0cacab-manifests\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.383769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.383650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ddca1d2-1117-442d-a537-26515c0cacab-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.383769 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.383705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ddca1d2-1117-442d-a537-26515c0cacab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.383893 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.383778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xd8\" (UniqueName: \"kubernetes.io/projected/2ddca1d2-1117-442d-a537-26515c0cacab-kube-api-access-w9xd8\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.484363 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.484340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xd8\" (UniqueName: \"kubernetes.io/projected/2ddca1d2-1117-442d-a537-26515c0cacab-kube-api-access-w9xd8\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.484451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.484377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ddca1d2-1117-442d-a537-26515c0cacab-tmp\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.484451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.484408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ddca1d2-1117-442d-a537-26515c0cacab-manifests\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.484451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.484423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ddca1d2-1117-442d-a537-26515c0cacab-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.484451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.484443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ddca1d2-1117-442d-a537-26515c0cacab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.484815 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.484793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ddca1d2-1117-442d-a537-26515c0cacab-manifests\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.486438 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.486413    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ddca1d2-1117-442d-a537-26515c0cacab-tmp\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.486744 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.486714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ddca1d2-1117-442d-a537-26515c0cacab-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.486817 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.486712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ddca1d2-1117-442d-a537-26515c0cacab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.492301 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.492280    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xd8\" (UniqueName: \"kubernetes.io/projected/2ddca1d2-1117-442d-a537-26515c0cacab-kube-api-access-w9xd8\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-c8f2c\" (UID: \"2ddca1d2-1117-442d-a537-26515c0cacab\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.572476 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.572431    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:27.852377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:27.852353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"]
Sep 17 23:14:27.854907 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:14:27.854873    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddca1d2_1117_442d_a537_26515c0cacab.slice/crio-7f16460e5e2747a2a934c9cef746c3aa5d977768b830aa22cbfa17a59586e1d0 WatchSource:0}: Error finding container 7f16460e5e2747a2a934c9cef746c3aa5d977768b830aa22cbfa17a59586e1d0: Status 404 returned error can't find the container with id 7f16460e5e2747a2a934c9cef746c3aa5d977768b830aa22cbfa17a59586e1d0
Sep 17 23:14:28.520587 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:28.520538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c" event={"ID":"2ddca1d2-1117-442d-a537-26515c0cacab","Type":"ContainerStarted","Data":"7f16460e5e2747a2a934c9cef746c3aa5d977768b830aa22cbfa17a59586e1d0"}
Sep 17 23:14:31.540957 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:31.540921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c" event={"ID":"2ddca1d2-1117-442d-a537-26515c0cacab","Type":"ContainerStarted","Data":"7de755c4e20efc47eb9968822cf1e3d9fc3b2b147622f77e3219d9e481d6d266"}
Sep 17 23:14:32.546872 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:32.546843    2559 generic.go:356] "Generic (PLEG): container finished" podID="2ddca1d2-1117-442d-a537-26515c0cacab" containerID="7de755c4e20efc47eb9968822cf1e3d9fc3b2b147622f77e3219d9e481d6d266" exitCode=0
Sep 17 23:14:32.547244 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:32.546933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c" event={"ID":"2ddca1d2-1117-442d-a537-26515c0cacab","Type":"ContainerDied","Data":"7de755c4e20efc47eb9968822cf1e3d9fc3b2b147622f77e3219d9e481d6d266"}
Sep 17 23:14:33.553795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:33.553758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c" event={"ID":"2ddca1d2-1117-442d-a537-26515c0cacab","Type":"ContainerStarted","Data":"a42048681cabffe2cd8914602e02c1c063b09aaad1f500f4aeaae9051f73a3b1"}
Sep 17 23:14:33.554225 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:33.553848    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:33.580519 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:33.580471    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c" podStartSLOduration=2.283712707 podStartE2EDuration="6.580456591s" totalImagesPullingTime="3.226615872s" totalInitContainerRuntime="1.070128012s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:33.578625281 +0000 UTC m=+516.052568267" watchObservedRunningTime="2026-09-17 23:14:33.580456591 +0000 UTC m=+516.054399574"
Sep 17 23:14:44.559470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:44.559399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-c8f2c"
Sep 17 23:14:45.142270 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.142236    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"]
Sep 17 23:14:45.145527 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.145511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.147670 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.147647    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 23:14:45.148461 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.148441    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 23:14:45.148529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.148446    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bmztk\""
Sep 17 23:14:45.148529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.148469    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 23:14:45.153759 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.153739    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"]
Sep 17 23:14:45.317618 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.317595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/546b24b5-7de7-4e83-8045-897fc2757973-manager-config\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.317701 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.317639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/546b24b5-7de7-4e83-8045-897fc2757973-cert\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.317701 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.317662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkt5\" (UniqueName: \"kubernetes.io/projected/546b24b5-7de7-4e83-8045-897fc2757973-kube-api-access-5fkt5\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.317701 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.317683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/546b24b5-7de7-4e83-8045-897fc2757973-metrics-cert\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.418568 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.418500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/546b24b5-7de7-4e83-8045-897fc2757973-cert\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.418568 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.418553    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkt5\" (UniqueName: \"kubernetes.io/projected/546b24b5-7de7-4e83-8045-897fc2757973-kube-api-access-5fkt5\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.418689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.418666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/546b24b5-7de7-4e83-8045-897fc2757973-metrics-cert\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.418778 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.418761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/546b24b5-7de7-4e83-8045-897fc2757973-manager-config\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.419364 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.419344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/546b24b5-7de7-4e83-8045-897fc2757973-manager-config\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.420936 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.420918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/546b24b5-7de7-4e83-8045-897fc2757973-cert\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.421005 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.420955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/546b24b5-7de7-4e83-8045-897fc2757973-metrics-cert\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.426358 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.426335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkt5\" (UniqueName: \"kubernetes.io/projected/546b24b5-7de7-4e83-8045-897fc2757973-kube-api-access-5fkt5\") pod \"lws-controller-manager-678fd6f85-dcq2d\" (UID: \"546b24b5-7de7-4e83-8045-897fc2757973\") " pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.455426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.455407    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:45.731558 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:45.731534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"]
Sep 17 23:14:45.734105 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:14:45.734076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546b24b5_7de7_4e83_8045_897fc2757973.slice/crio-857b4c3844a60101ea3ed83b7e55f29b6723dd92e895bf66ed7b11426115baf1 WatchSource:0}: Error finding container 857b4c3844a60101ea3ed83b7e55f29b6723dd92e895bf66ed7b11426115baf1: Status 404 returned error can't find the container with id 857b4c3844a60101ea3ed83b7e55f29b6723dd92e895bf66ed7b11426115baf1
Sep 17 23:14:46.612544 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.612503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d" event={"ID":"546b24b5-7de7-4e83-8045-897fc2757973","Type":"ContainerStarted","Data":"857b4c3844a60101ea3ed83b7e55f29b6723dd92e895bf66ed7b11426115baf1"}
Sep 17 23:14:46.753141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.753108    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"]
Sep 17 23:14:46.756435 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.756416    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:46.760189 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.760169    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 23:14:46.761236 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.761195    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 23:14:46.762188 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.762171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hvlzs\""
Sep 17 23:14:46.776571 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.776538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"]
Sep 17 23:14:46.929836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.929751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:46.929836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.929813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tmp\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:46.930043 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:46.929842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmt9\" (UniqueName: \"kubernetes.io/projected/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-kube-api-access-xwmt9\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.030839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.030806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.030998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.030855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tmp\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.030998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.030878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmt9\" (UniqueName: \"kubernetes.io/projected/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-kube-api-access-xwmt9\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.030998 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:14:47.030970    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 17 23:14:47.031111 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:14:47.031040    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs podName:4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:47.531023316 +0000 UTC m=+530.004966282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs") pod "kube-auth-proxy-d4c67b97b-b7wws" (UID: "4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6") : secret "kube-auth-proxy-tls" not found
Sep 17 23:14:47.033272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.033248    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tmp\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.043483 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.043460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmt9\" (UniqueName: \"kubernetes.io/projected/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-kube-api-access-xwmt9\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.535528 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.535449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:47.535668 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:14:47.535611    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 17 23:14:47.535705 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:14:47.535677    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs podName:4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:48.535661708 +0000 UTC m=+531.009604670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs") pod "kube-auth-proxy-d4c67b97b-b7wws" (UID: "4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6") : secret "kube-auth-proxy-tls" not found
Sep 17 23:14:47.619056 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.619026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d" event={"ID":"546b24b5-7de7-4e83-8045-897fc2757973","Type":"ContainerStarted","Data":"587c65f1c501e4225657c204d0e68b79eadc46ca24c5b0891814341d786f5b8d"}
Sep 17 23:14:47.619201 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.619121    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:14:47.653874 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:47.653830    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d" podStartSLOduration=1.18490524 podStartE2EDuration="2.653816889s" totalImagesPullingTime="1.468911649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:47.651560337 +0000 UTC m=+530.125503330" watchObservedRunningTime="2026-09-17 23:14:47.653816889 +0000 UTC m=+530.127759870"
Sep 17 23:14:48.543207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:48.543175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:48.545472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:48.545445    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-b7wws\" (UID: \"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:48.567385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:48.567353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"
Sep 17 23:14:48.839141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:48.839073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws"]
Sep 17 23:14:48.843081 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:14:48.843055    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab03d10_ee8b_45ac_ae6f_d2a2a7283ec6.slice/crio-c9a2a6b9784112a8a02c1858d06c86f8f1a2e822db55615b3a6d7c4f96d68a4f WatchSource:0}: Error finding container c9a2a6b9784112a8a02c1858d06c86f8f1a2e822db55615b3a6d7c4f96d68a4f: Status 404 returned error can't find the container with id c9a2a6b9784112a8a02c1858d06c86f8f1a2e822db55615b3a6d7c4f96d68a4f
Sep 17 23:14:49.632200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:49.632163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws" event={"ID":"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6","Type":"ContainerStarted","Data":"c9a2a6b9784112a8a02c1858d06c86f8f1a2e822db55615b3a6d7c4f96d68a4f"}
Sep 17 23:14:51.642475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:51.642434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws" event={"ID":"4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6","Type":"ContainerStarted","Data":"b98d642e8c5ccf399c6b214777e6b6afdbe372405739e3acef1f997228ef7f60"}
Sep 17 23:14:51.659419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:51.659373    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-b7wws" podStartSLOduration=3.076040551 podStartE2EDuration="5.659358766s" totalImagesPullingTime="2.583318215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:51.65784094 +0000 UTC m=+534.131783921" watchObservedRunningTime="2026-09-17 23:14:51.659358766 +0000 UTC m=+534.133301748"
Sep 17 23:14:58.626633 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:14:58.626603    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-678fd6f85-dcq2d"
Sep 17 23:15:59.560318 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:15:59.560288    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:15:59.561742 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:15:59.561703    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:15:59.569177 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:15:59.569159    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:15:59.570213 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:15:59.570190    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:16:31.209667 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.209635    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"]
Sep 17 23:16:31.213061 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.213039    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.215882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.215863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 23:16:31.216004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.215889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 23:16:31.216004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.215895    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-97v8q\""
Sep 17 23:16:31.216572 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.216557    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 23:16:31.216662 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.216586    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 23:16:31.223284 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.223264    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"]
Sep 17 23:16:31.338785 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.338760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fc0bfad-448c-4b7b-adf9-91fabb127593-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.338880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.338795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fc0bfad-448c-4b7b-adf9-91fabb127593-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.338880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.338871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwrw\" (UniqueName: \"kubernetes.io/projected/4fc0bfad-448c-4b7b-adf9-91fabb127593-kube-api-access-bpwrw\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.439810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.439785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fc0bfad-448c-4b7b-adf9-91fabb127593-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.439902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.439859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwrw\" (UniqueName: \"kubernetes.io/projected/4fc0bfad-448c-4b7b-adf9-91fabb127593-kube-api-access-bpwrw\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.439902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.439896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fc0bfad-448c-4b7b-adf9-91fabb127593-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.440529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.440513    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fc0bfad-448c-4b7b-adf9-91fabb127593-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.442028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.442010    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fc0bfad-448c-4b7b-adf9-91fabb127593-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.447689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.447666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwrw\" (UniqueName: \"kubernetes.io/projected/4fc0bfad-448c-4b7b-adf9-91fabb127593-kube-api-access-bpwrw\") pod \"kuadrant-console-plugin-fbdd4f445-pjlms\" (UID: \"4fc0bfad-448c-4b7b-adf9-91fabb127593\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.522505 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.522453    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"
Sep 17 23:16:31.793065 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.793039    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms"]
Sep 17 23:16:31.795505 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:16:31.795473    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fc0bfad_448c_4b7b_adf9_91fabb127593.slice/crio-3044095dbd20820ec632d7141e73db89b6e0fa1b26d89c4b21ef6e8298668e13 WatchSource:0}: Error finding container 3044095dbd20820ec632d7141e73db89b6e0fa1b26d89c4b21ef6e8298668e13: Status 404 returned error can't find the container with id 3044095dbd20820ec632d7141e73db89b6e0fa1b26d89c4b21ef6e8298668e13
Sep 17 23:16:31.796824 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:31.796807    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:16:32.072560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:32.072493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms" event={"ID":"4fc0bfad-448c-4b7b-adf9-91fabb127593","Type":"ContainerStarted","Data":"3044095dbd20820ec632d7141e73db89b6e0fa1b26d89c4b21ef6e8298668e13"}
Sep 17 23:16:37.099903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:37.099867    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms" event={"ID":"4fc0bfad-448c-4b7b-adf9-91fabb127593","Type":"ContainerStarted","Data":"995c3b72609cf3c6cc75d0b2075d52dff0554b085b356bff1f014d089cda0082"}
Sep 17 23:16:37.114802 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:16:37.114761    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pjlms" podStartSLOduration=1.483104354 podStartE2EDuration="6.114747074s" totalImagesPullingTime="4.63164272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:37.113409272 +0000 UTC m=+639.587352263" watchObservedRunningTime="2026-09-17 23:16:37.114747074 +0000 UTC m=+639.588690047"
Sep 17 23:17:28.108922 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.108885    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:17:28.112283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.112265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.114594 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.114574    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 23:17:28.120709 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.120687    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:17:28.146767 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.146741    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:17:28.175192 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.175172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbgg\" (UniqueName: \"kubernetes.io/projected/3b4115b4-71b1-46d2-be35-c16dfdb14a60-kube-api-access-5sbgg\") pod \"limitador-limitador-84994f7ffc-qq8l2\" (UID: \"3b4115b4-71b1-46d2-be35-c16dfdb14a60\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.175257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.175200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b4115b4-71b1-46d2-be35-c16dfdb14a60-config-file\") pod \"limitador-limitador-84994f7ffc-qq8l2\" (UID: \"3b4115b4-71b1-46d2-be35-c16dfdb14a60\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.275938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.275912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbgg\" (UniqueName: \"kubernetes.io/projected/3b4115b4-71b1-46d2-be35-c16dfdb14a60-kube-api-access-5sbgg\") pod \"limitador-limitador-84994f7ffc-qq8l2\" (UID: \"3b4115b4-71b1-46d2-be35-c16dfdb14a60\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.276027 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.275943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b4115b4-71b1-46d2-be35-c16dfdb14a60-config-file\") pod \"limitador-limitador-84994f7ffc-qq8l2\" (UID: \"3b4115b4-71b1-46d2-be35-c16dfdb14a60\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.276505 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.276487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b4115b4-71b1-46d2-be35-c16dfdb14a60-config-file\") pod \"limitador-limitador-84994f7ffc-qq8l2\" (UID: \"3b4115b4-71b1-46d2-be35-c16dfdb14a60\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.284364 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.284336    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbgg\" (UniqueName: \"kubernetes.io/projected/3b4115b4-71b1-46d2-be35-c16dfdb14a60-kube-api-access-5sbgg\") pod \"limitador-limitador-84994f7ffc-qq8l2\" (UID: \"3b4115b4-71b1-46d2-be35-c16dfdb14a60\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.422550 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.422482    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:28.432819 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.432792    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-lls99"]
Sep 17 23:17:28.436354 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.436333    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:17:28.438564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.438543    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cmz9j\""
Sep 17 23:17:28.445291 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.445264    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lls99"]
Sep 17 23:17:28.477337 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.477266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcgj\" (UniqueName: \"kubernetes.io/projected/273c3600-d767-40af-9b62-e4aaa37d2713-kube-api-access-nrcgj\") pod \"authorino-59c974bd48-lls99\" (UID: \"273c3600-d767-40af-9b62-e4aaa37d2713\") " pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:17:28.578764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.578714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcgj\" (UniqueName: \"kubernetes.io/projected/273c3600-d767-40af-9b62-e4aaa37d2713-kube-api-access-nrcgj\") pod \"authorino-59c974bd48-lls99\" (UID: \"273c3600-d767-40af-9b62-e4aaa37d2713\") " pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:17:28.586703 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.586680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcgj\" (UniqueName: \"kubernetes.io/projected/273c3600-d767-40af-9b62-e4aaa37d2713-kube-api-access-nrcgj\") pod \"authorino-59c974bd48-lls99\" (UID: \"273c3600-d767-40af-9b62-e4aaa37d2713\") " pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:17:28.695934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.695912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:17:28.698393 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:17:28.698371    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4115b4_71b1_46d2_be35_c16dfdb14a60.slice/crio-58b0f37c1790df6635c7e379fdc1a92787b4ab6c03b7bf1649c77f96f9f79258 WatchSource:0}: Error finding container 58b0f37c1790df6635c7e379fdc1a92787b4ab6c03b7bf1649c77f96f9f79258: Status 404 returned error can't find the container with id 58b0f37c1790df6635c7e379fdc1a92787b4ab6c03b7bf1649c77f96f9f79258
Sep 17 23:17:28.757177 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:28.757158    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:17:29.023730 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:29.023692    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lls99"]
Sep 17 23:17:29.026126 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:17:29.026101    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod273c3600_d767_40af_9b62_e4aaa37d2713.slice/crio-47e3a2c439a94284125c055c56afa6d67ab4bb473fa27463c1337570bbae2ab2 WatchSource:0}: Error finding container 47e3a2c439a94284125c055c56afa6d67ab4bb473fa27463c1337570bbae2ab2: Status 404 returned error can't find the container with id 47e3a2c439a94284125c055c56afa6d67ab4bb473fa27463c1337570bbae2ab2
Sep 17 23:17:29.344304 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:29.344208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lls99" event={"ID":"273c3600-d767-40af-9b62-e4aaa37d2713","Type":"ContainerStarted","Data":"47e3a2c439a94284125c055c56afa6d67ab4bb473fa27463c1337570bbae2ab2"}
Sep 17 23:17:29.345969 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:29.345901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2" event={"ID":"3b4115b4-71b1-46d2-be35-c16dfdb14a60","Type":"ContainerStarted","Data":"58b0f37c1790df6635c7e379fdc1a92787b4ab6c03b7bf1649c77f96f9f79258"}
Sep 17 23:17:32.363382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:32.363337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lls99" event={"ID":"273c3600-d767-40af-9b62-e4aaa37d2713","Type":"ContainerStarted","Data":"2f693e8383f4e532527c8a58daa077438e06f38b2b3a26ecbb91b45b3d1ad513"}
Sep 17 23:17:32.365314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:32.365284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2" event={"ID":"3b4115b4-71b1-46d2-be35-c16dfdb14a60","Type":"ContainerStarted","Data":"f13452f41fb46550dfbaf7c3f79429966fcbc28df1d8a20370bf61636ac672c5"}
Sep 17 23:17:32.365433 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:32.365374    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:17:32.379031 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:32.378981    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-lls99" podStartSLOduration=2.01956612 podStartE2EDuration="4.378965305s" totalImagesPullingTime="2.359399185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:32.376863322 +0000 UTC m=+694.850806323" watchObservedRunningTime="2026-09-17 23:17:32.378965305 +0000 UTC m=+694.852908287"
Sep 17 23:17:32.399377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:32.399334    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2" podStartSLOduration=1.718768173 podStartE2EDuration="4.399318383s" totalImagesPullingTime="2.68055021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:32.397676699 +0000 UTC m=+694.871619700" watchObservedRunningTime="2026-09-17 23:17:32.399318383 +0000 UTC m=+694.873261386"
Sep 17 23:17:43.371523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:17:43.371437    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"
Sep 17 23:18:00.341155 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.341108    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lls99"]
Sep 17 23:18:00.341841 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.341287    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-lls99" podUID="273c3600-d767-40af-9b62-e4aaa37d2713" containerName="authorino" containerID="cri-o://2f693e8383f4e532527c8a58daa077438e06f38b2b3a26ecbb91b45b3d1ad513" gracePeriod=30
Sep 17 23:18:00.508115 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.508087    2559 generic.go:356] "Generic (PLEG): container finished" podID="273c3600-d767-40af-9b62-e4aaa37d2713" containerID="2f693e8383f4e532527c8a58daa077438e06f38b2b3a26ecbb91b45b3d1ad513" exitCode=0
Sep 17 23:18:00.508241 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.508124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lls99" event={"ID":"273c3600-d767-40af-9b62-e4aaa37d2713","Type":"ContainerDied","Data":"2f693e8383f4e532527c8a58daa077438e06f38b2b3a26ecbb91b45b3d1ad513"}
Sep 17 23:18:00.575900 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.575879    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:18:00.722599 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.722570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrcgj\" (UniqueName: \"kubernetes.io/projected/273c3600-d767-40af-9b62-e4aaa37d2713-kube-api-access-nrcgj\") pod \"273c3600-d767-40af-9b62-e4aaa37d2713\" (UID: \"273c3600-d767-40af-9b62-e4aaa37d2713\") "
Sep 17 23:18:00.724832 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.724802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273c3600-d767-40af-9b62-e4aaa37d2713-kube-api-access-nrcgj" (OuterVolumeSpecName: "kube-api-access-nrcgj") pod "273c3600-d767-40af-9b62-e4aaa37d2713" (UID: "273c3600-d767-40af-9b62-e4aaa37d2713"). InnerVolumeSpecName "kube-api-access-nrcgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:18:00.823140 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:00.823116    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcgj\" (UniqueName: \"kubernetes.io/projected/273c3600-d767-40af-9b62-e4aaa37d2713-kube-api-access-nrcgj\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:18:01.514014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:01.513983    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lls99"
Sep 17 23:18:01.514481 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:01.514022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lls99" event={"ID":"273c3600-d767-40af-9b62-e4aaa37d2713","Type":"ContainerDied","Data":"47e3a2c439a94284125c055c56afa6d67ab4bb473fa27463c1337570bbae2ab2"}
Sep 17 23:18:01.514481 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:01.514085    2559 scope.go:117] "RemoveContainer" containerID="2f693e8383f4e532527c8a58daa077438e06f38b2b3a26ecbb91b45b3d1ad513"
Sep 17 23:18:01.535320 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:01.535292    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lls99"]
Sep 17 23:18:01.540607 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:01.540584    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lls99"]
Sep 17 23:18:01.638386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:01.638357    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="273c3600-d767-40af-9b62-e4aaa37d2713" path="/var/lib/kubelet/pods/273c3600-d767-40af-9b62-e4aaa37d2713/volumes"
Sep 17 23:18:42.536773 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.536736    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"]
Sep 17 23:18:42.537217 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.537052    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273c3600-d767-40af-9b62-e4aaa37d2713" containerName="authorino"
Sep 17 23:18:42.537217 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.537063    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="273c3600-d767-40af-9b62-e4aaa37d2713" containerName="authorino"
Sep 17 23:18:42.537217 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.537135    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="273c3600-d767-40af-9b62-e4aaa37d2713" containerName="authorino"
Sep 17 23:18:42.540122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.540106    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.542286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.542263    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wlnsl\""
Sep 17 23:18:42.542417 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.542301    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 23:18:42.550094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.550067    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"]
Sep 17 23:18:42.589088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.589068    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:18:42.627563 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.627542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373051c0-27e6-4947-ab58-e13b1a9182e4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-65bws\" (UID: \"373051c0-27e6-4947-ab58-e13b1a9182e4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.627653 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.627610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrwg\" (UniqueName: \"kubernetes.io/projected/373051c0-27e6-4947-ab58-e13b1a9182e4-kube-api-access-9mrwg\") pod \"payload-pre-processing-59fbc47bf8-65bws\" (UID: \"373051c0-27e6-4947-ab58-e13b1a9182e4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.728226 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.728196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373051c0-27e6-4947-ab58-e13b1a9182e4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-65bws\" (UID: \"373051c0-27e6-4947-ab58-e13b1a9182e4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.728314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.728269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrwg\" (UniqueName: \"kubernetes.io/projected/373051c0-27e6-4947-ab58-e13b1a9182e4-kube-api-access-9mrwg\") pod \"payload-pre-processing-59fbc47bf8-65bws\" (UID: \"373051c0-27e6-4947-ab58-e13b1a9182e4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.728847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.728830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373051c0-27e6-4947-ab58-e13b1a9182e4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-65bws\" (UID: \"373051c0-27e6-4947-ab58-e13b1a9182e4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.739151 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.739128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrwg\" (UniqueName: \"kubernetes.io/projected/373051c0-27e6-4947-ab58-e13b1a9182e4-kube-api-access-9mrwg\") pod \"payload-pre-processing-59fbc47bf8-65bws\" (UID: \"373051c0-27e6-4947-ab58-e13b1a9182e4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:42.852123 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:42.852072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:43.123173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:43.123146    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"]
Sep 17 23:18:43.127203 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:18:43.127176    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373051c0_27e6_4947_ab58_e13b1a9182e4.slice/crio-d5ba9060a4758a0bd8d2f5d10fd56c6b6068dd3ffc6c42c3eb6a69d5d98af33d WatchSource:0}: Error finding container d5ba9060a4758a0bd8d2f5d10fd56c6b6068dd3ffc6c42c3eb6a69d5d98af33d: Status 404 returned error can't find the container with id d5ba9060a4758a0bd8d2f5d10fd56c6b6068dd3ffc6c42c3eb6a69d5d98af33d
Sep 17 23:18:43.712482 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:43.712441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws" event={"ID":"373051c0-27e6-4947-ab58-e13b1a9182e4","Type":"ContainerStarted","Data":"d5ba9060a4758a0bd8d2f5d10fd56c6b6068dd3ffc6c42c3eb6a69d5d98af33d"}
Sep 17 23:18:45.724787 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:45.724753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws" event={"ID":"373051c0-27e6-4947-ab58-e13b1a9182e4","Type":"ContainerStarted","Data":"6f22b958c3007328a9a00df1c88e033ac613b854bcf03df5fdf636e723991c16"}
Sep 17 23:18:45.725164 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:45.724829    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:18:45.738332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:45.738276    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws" podStartSLOduration=1.253572417 podStartE2EDuration="3.73825758s" totalImagesPullingTime="2.484685163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:45.737394984 +0000 UTC m=+768.211337969" watchObservedRunningTime="2026-09-17 23:18:45.73825758 +0000 UTC m=+768.212200564"
Sep 17 23:18:56.731095 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:18:56.731066    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-65bws"
Sep 17 23:19:04.142900 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:04.142857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:19:07.726919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:07.726879    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:19:24.928479 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.928443    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"]
Sep 17 23:19:24.937522 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.937495    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:24.940788 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.940757    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"]
Sep 17 23:19:24.941000 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.940787    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 23:19:24.941126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.940787    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 23:19:24.941126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.940787    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jks5l\""
Sep 17 23:19:24.941126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:24.940816    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 23:19:25.056460 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.056433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.056575 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.056469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b26bfa-6959-4baf-ad76-89ecd0431c13-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.056575 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.056497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t56v\" (UniqueName: \"kubernetes.io/projected/e0b26bfa-6959-4baf-ad76-89ecd0431c13-kube-api-access-6t56v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.056684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.056578    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.056684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.056633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.056796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.056702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157112 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t56v\" (UniqueName: \"kubernetes.io/projected/e0b26bfa-6959-4baf-ad76-89ecd0431c13-kube-api-access-6t56v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157244 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157244 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157353 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157353 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157353 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b26bfa-6959-4baf-ad76-89ecd0431c13-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157538 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.157647 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.157619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.159695 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.159665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b26bfa-6959-4baf-ad76-89ecd0431c13-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.160004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.159983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b26bfa-6959-4baf-ad76-89ecd0431c13-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.164872 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.164853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t56v\" (UniqueName: \"kubernetes.io/projected/e0b26bfa-6959-4baf-ad76-89ecd0431c13-kube-api-access-6t56v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79\" (UID: \"e0b26bfa-6959-4baf-ad76-89ecd0431c13\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.249062 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.249040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:25.519203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.519130    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"]
Sep 17 23:19:25.521659 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:19:25.521632    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b26bfa_6959_4baf_ad76_89ecd0431c13.slice/crio-320523cd541a770e0a0e3fb97a056db181bdc45b2ca4e0f8bcc9453df068b20d WatchSource:0}: Error finding container 320523cd541a770e0a0e3fb97a056db181bdc45b2ca4e0f8bcc9453df068b20d: Status 404 returned error can't find the container with id 320523cd541a770e0a0e3fb97a056db181bdc45b2ca4e0f8bcc9453df068b20d
Sep 17 23:19:25.924330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.924253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79" event={"ID":"e0b26bfa-6959-4baf-ad76-89ecd0431c13","Type":"ContainerStarted","Data":"320523cd541a770e0a0e3fb97a056db181bdc45b2ca4e0f8bcc9453df068b20d"}
Sep 17 23:19:25.932936 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:25.932909    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:19:26.931189 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:26.931095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79" event={"ID":"e0b26bfa-6959-4baf-ad76-89ecd0431c13","Type":"ContainerStarted","Data":"c747cf1c3cdf480888be4479fac90a7d644d857304e3f3112e2bb02b6674b5c7"}
Sep 17 23:19:26.947181 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:26.947136    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79" podStartSLOduration=1.9549875110000001 podStartE2EDuration="2.947091716s" totalImagesPullingTime="992.104205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:26.94642951 +0000 UTC m=+809.420372496" watchObservedRunningTime="2026-09-17 23:19:26.947091716 +0000 UTC m=+809.421034703"
Sep 17 23:19:33.933125 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:33.933085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:19:35.249458 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:35.249421    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:35.249896 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:35.249485    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:35.262350 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:35.262319    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:35.986488 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:35.986458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79"
Sep 17 23:19:41.926783 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:41.926744    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:19:46.630696 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.630661    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"]
Sep 17 23:19:46.634110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.634083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.636323 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.636298    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 23:19:46.643111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.643085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"]
Sep 17 23:19:46.713837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.713812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22feb1fb-5506-49fb-8878-3a6104a0a0e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.713934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.713866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.713934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.713896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.714107 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.714092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.714144 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.714121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.714181 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.714143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm4z\" (UniqueName: \"kubernetes.io/projected/22feb1fb-5506-49fb-8878-3a6104a0a0e2-kube-api-access-srm4z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22feb1fb-5506-49fb-8878-3a6104a0a0e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm4z\" (UniqueName: \"kubernetes.io/projected/22feb1fb-5506-49fb-8878-3a6104a0a0e2-kube-api-access-srm4z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.815955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.815889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.817582 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.817561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22feb1fb-5506-49fb-8878-3a6104a0a0e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.817684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.817665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22feb1fb-5506-49fb-8878-3a6104a0a0e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.822337 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.822316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm4z\" (UniqueName: \"kubernetes.io/projected/22feb1fb-5506-49fb-8878-3a6104a0a0e2-kube-api-access-srm4z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kww9r\" (UID: \"22feb1fb-5506-49fb-8878-3a6104a0a0e2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:46.946288 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:46.946230    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:47.220286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:47.220258    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"]
Sep 17 23:19:47.222627 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:19:47.222603    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22feb1fb_5506_49fb_8878_3a6104a0a0e2.slice/crio-ab185979a692fbcd9de9c98ac840286874434b8775bbac0b00cbefc16a581aad WatchSource:0}: Error finding container ab185979a692fbcd9de9c98ac840286874434b8775bbac0b00cbefc16a581aad: Status 404 returned error can't find the container with id ab185979a692fbcd9de9c98ac840286874434b8775bbac0b00cbefc16a581aad
Sep 17 23:19:48.035953 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:48.035914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r" event={"ID":"22feb1fb-5506-49fb-8878-3a6104a0a0e2","Type":"ContainerStarted","Data":"5ab9405bbff16fb77b64a5c9c078b8abce49a972b40b3368af0737653f5b96d2"}
Sep 17 23:19:48.036330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:48.035978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r" event={"ID":"22feb1fb-5506-49fb-8878-3a6104a0a0e2","Type":"ContainerStarted","Data":"ab185979a692fbcd9de9c98ac840286874434b8775bbac0b00cbefc16a581aad"}
Sep 17 23:19:48.053201 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:48.053155    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r" podStartSLOduration=1.85853578 podStartE2EDuration="2.053139204s" totalImagesPullingTime="194.603424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:48.051248756 +0000 UTC m=+830.525191739" watchObservedRunningTime="2026-09-17 23:19:48.053139204 +0000 UTC m=+830.527082187"
Sep 17 23:19:48.432329 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:48.432261    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:19:54.724887 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.724842    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"]
Sep 17 23:19:54.728674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.728647    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.730882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.730857    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 23:19:54.737801 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.737777    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"]
Sep 17 23:19:54.780840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.780814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.780961 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.780911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad583cf3-dc68-4db9-930e-8fe032107dd8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.781036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.780967    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw88h\" (UniqueName: \"kubernetes.io/projected/ad583cf3-dc68-4db9-930e-8fe032107dd8-kube-api-access-cw88h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.781036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.781002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.781036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.781030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.781205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.781047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw88h\" (UniqueName: \"kubernetes.io/projected/ad583cf3-dc68-4db9-930e-8fe032107dd8-kube-api-access-cw88h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881677 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad583cf3-dc68-4db9-930e-8fe032107dd8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881751 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.881945 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.881925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.883710 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.883682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad583cf3-dc68-4db9-930e-8fe032107dd8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.883971 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.883954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad583cf3-dc68-4db9-930e-8fe032107dd8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:54.888778 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:54.888745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw88h\" (UniqueName: \"kubernetes.io/projected/ad583cf3-dc68-4db9-930e-8fe032107dd8-kube-api-access-cw88h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-svp2b\" (UID: \"ad583cf3-dc68-4db9-930e-8fe032107dd8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:55.040329 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:55.040271    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:19:55.310150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:55.310063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"]
Sep 17 23:19:55.312313 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:19:55.312287    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad583cf3_dc68_4db9_930e_8fe032107dd8.slice/crio-6c8ef3750896d52d2ba3f58c3f68863606edef30b97e69a1e10f01b5380a8d46 WatchSource:0}: Error finding container 6c8ef3750896d52d2ba3f58c3f68863606edef30b97e69a1e10f01b5380a8d46: Status 404 returned error can't find the container with id 6c8ef3750896d52d2ba3f58c3f68863606edef30b97e69a1e10f01b5380a8d46
Sep 17 23:19:56.076386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:56.076344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b" event={"ID":"ad583cf3-dc68-4db9-930e-8fe032107dd8","Type":"ContainerStarted","Data":"4e6ff73519945ff176501e2ef4188e71d27a616f10e60452c2f34bbbea00c70a"}
Sep 17 23:19:56.076386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:56.076393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b" event={"ID":"ad583cf3-dc68-4db9-930e-8fe032107dd8","Type":"ContainerStarted","Data":"6c8ef3750896d52d2ba3f58c3f68863606edef30b97e69a1e10f01b5380a8d46"}
Sep 17 23:19:56.093800 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:56.093755    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b" podStartSLOduration=1.904091757 podStartE2EDuration="2.09373827s" totalImagesPullingTime="189.646513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:56.09165805 +0000 UTC m=+838.565601033" watchObservedRunningTime="2026-09-17 23:19:56.09373827 +0000 UTC m=+838.567681244"
Sep 17 23:19:56.946780 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:56.946744    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:56.946967 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:56.946810    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:56.959477 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:56.959449    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:57.092103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:57.092080    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kww9r"
Sep 17 23:19:57.137413 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:19:57.137385    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:20:05.041048 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:05.040999    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:20:05.041666 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:05.041092    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:20:05.053816 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:05.053788    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:20:05.138731 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:05.138695    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-svp2b"
Sep 17 23:20:59.593210 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:59.593179    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:20:59.594218 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:59.594197    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:20:59.600939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:59.600920    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:20:59.601866 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:20:59.601845    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:21:35.921141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:35.921100    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7861-719527"]
Sep 17 23:21:35.924837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:35.924814    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:35.927185 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:35.927166    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6p2p\""
Sep 17 23:21:35.930168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:35.930141    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-719527"]
Sep 17 23:21:36.078308 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.078281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcsc\" (UniqueName: \"kubernetes.io/projected/6e43f6ab-8359-46a2-be8c-787253660640-kube-api-access-tqcsc\") pod \"test-curl-7861-719527\" (UID: \"6e43f6ab-8359-46a2-be8c-787253660640\") " pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:36.179580 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.179514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcsc\" (UniqueName: \"kubernetes.io/projected/6e43f6ab-8359-46a2-be8c-787253660640-kube-api-access-tqcsc\") pod \"test-curl-7861-719527\" (UID: \"6e43f6ab-8359-46a2-be8c-787253660640\") " pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:36.188211 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.188177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcsc\" (UniqueName: \"kubernetes.io/projected/6e43f6ab-8359-46a2-be8c-787253660640-kube-api-access-tqcsc\") pod \"test-curl-7861-719527\" (UID: \"6e43f6ab-8359-46a2-be8c-787253660640\") " pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:36.235049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.235024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:36.507791 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.507765    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-719527"]
Sep 17 23:21:36.510227 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:36.510196    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e43f6ab_8359_46a2_be8c_787253660640.slice/crio-170c313e0c8bd6411175cb4785475018c46ed93ba3e9e2100a9c80926855f5c1 WatchSource:0}: Error finding container 170c313e0c8bd6411175cb4785475018c46ed93ba3e9e2100a9c80926855f5c1: Status 404 returned error can't find the container with id 170c313e0c8bd6411175cb4785475018c46ed93ba3e9e2100a9c80926855f5c1
Sep 17 23:21:36.512147 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.512123    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:21:36.581430 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:36.581404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-719527" event={"ID":"6e43f6ab-8359-46a2-be8c-787253660640","Type":"ContainerStarted","Data":"170c313e0c8bd6411175cb4785475018c46ed93ba3e9e2100a9c80926855f5c1"}
Sep 17 23:21:37.858072 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:37.858037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:21:39.438891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.438853    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"]
Sep 17 23:21:39.442306 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.442289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.444700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.444630    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-861522-dockercfg-lt8ws\""
Sep 17 23:21:39.444700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.444666    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-861522\""
Sep 17 23:21:39.453172 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.453144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"]
Sep 17 23:21:39.456164 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.456144    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"]
Sep 17 23:21:39.459344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.459325    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.472684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.472662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"]
Sep 17 23:21:39.602617 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.602580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-719527" event={"ID":"6e43f6ab-8359-46a2-be8c-787253660640","Type":"ContainerStarted","Data":"27adbfc9522c2d728387ee811d244bf8c134c49e30bae6d57a9f59eb8c0b0e4b"}
Sep 17 23:21:39.609900 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.609873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.610191 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.609911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7df89-adda-4ccd-b183-1e91d10023a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.610191 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.609964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxpk\" (UniqueName: \"kubernetes.io/projected/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-kube-api-access-2vxpk\") pod \"payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.610191 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.609992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzbv\" (UniqueName: \"kubernetes.io/projected/39d7df89-adda-4ccd-b183-1e91d10023a0-kube-api-access-kjzbv\") pod \"payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.619223 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.619184    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7861-719527" podStartSLOduration=2.487766055 podStartE2EDuration="4.61916929s" totalImagesPullingTime="2.131403235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:39.617245136 +0000 UTC m=+942.091188120" watchObservedRunningTime="2026-09-17 23:21:39.61916929 +0000 UTC m=+942.093112314"
Sep 17 23:21:39.711280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.711248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.711424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.711312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7df89-adda-4ccd-b183-1e91d10023a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.711490 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.711423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxpk\" (UniqueName: \"kubernetes.io/projected/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-kube-api-access-2vxpk\") pod \"payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.711549 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.711510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzbv\" (UniqueName: \"kubernetes.io/projected/39d7df89-adda-4ccd-b183-1e91d10023a0-kube-api-access-kjzbv\") pod \"payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.712200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.712178    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.712326 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.712201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7df89-adda-4ccd-b183-1e91d10023a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.725378 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.725354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxpk\" (UniqueName: \"kubernetes.io/projected/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-kube-api-access-2vxpk\") pod \"payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:39.728490 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.728466    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzbv\" (UniqueName: \"kubernetes.io/projected/39d7df89-adda-4ccd-b183-1e91d10023a0-kube-api-access-kjzbv\") pod \"payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.753294 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.753269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:39.768485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:39.768466    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:40.032889 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.032852    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"]
Sep 17 23:21:40.036492 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:40.036464    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d7df89_adda_4ccd_b183_1e91d10023a0.slice/crio-42f05485657ad457a7b7404758092b4c0fb057d588a8f83565d666402bef6060 WatchSource:0}: Error finding container 42f05485657ad457a7b7404758092b4c0fb057d588a8f83565d666402bef6060: Status 404 returned error can't find the container with id 42f05485657ad457a7b7404758092b4c0fb057d588a8f83565d666402bef6060
Sep 17 23:21:40.063844 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.063706    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"]
Sep 17 23:21:40.066853 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:40.066827    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a1c5c8_817c_4d8c_bd12_b8b69fea2610.slice/crio-a2f2cd3b2fbad025686aced8cb8e741bcdaf29c878496e71507f9320ac0919ae WatchSource:0}: Error finding container a2f2cd3b2fbad025686aced8cb8e741bcdaf29c878496e71507f9320ac0919ae: Status 404 returned error can't find the container with id a2f2cd3b2fbad025686aced8cb8e741bcdaf29c878496e71507f9320ac0919ae
Sep 17 23:21:40.243330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.243295    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7861-719527"]
Sep 17 23:21:40.253424 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.253126    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7861-719527"]
Sep 17 23:21:40.593505 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.593475    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"]
Sep 17 23:21:40.593874 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.593850    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e43f6ab-8359-46a2-be8c-787253660640" containerName="test-curl-7861-719527"
Sep 17 23:21:40.593874 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.593862    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e43f6ab-8359-46a2-be8c-787253660640" containerName="test-curl-7861-719527"
Sep 17 23:21:40.593945 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.593929    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e43f6ab-8359-46a2-be8c-787253660640" containerName="test-curl-7861-719527"
Sep 17 23:21:40.596972 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.596953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.599492 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.599472    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-983490-dockercfg-zs49f\""
Sep 17 23:21:40.599659 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.599643    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-983490\""
Sep 17 23:21:40.609081 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.609063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"]
Sep 17 23:21:40.610290 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.610266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp" event={"ID":"39d7df89-adda-4ccd-b183-1e91d10023a0","Type":"ContainerStarted","Data":"d1441a4330f6811d9fc2dae1b52f5dd1f2abc87546ad6f4709929c10226bdcb8"}
Sep 17 23:21:40.610384 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.610315    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:40.610384 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.610333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp" event={"ID":"39d7df89-adda-4ccd-b183-1e91d10023a0","Type":"ContainerStarted","Data":"42f05485657ad457a7b7404758092b4c0fb057d588a8f83565d666402bef6060"}
Sep 17 23:21:40.612163 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.612139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b" event={"ID":"02a1c5c8-817c-4d8c-bd12-b8b69fea2610","Type":"ContainerStarted","Data":"68d58a2e74c0eb50aba638607b166e4084a4f1142913b15614cd153b8d6082c5"}
Sep 17 23:21:40.612258 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.612184    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b" event={"ID":"02a1c5c8-817c-4d8c-bd12-b8b69fea2610","Type":"ContainerStarted","Data":"a2f2cd3b2fbad025686aced8cb8e741bcdaf29c878496e71507f9320ac0919ae"}
Sep 17 23:21:40.612325 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.612265    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:40.612426 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:21:40.612407    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7861-719527" container="test-curl-7861-719527"
Sep 17 23:21:40.631234 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.631209    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"]
Sep 17 23:21:40.635052 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.635034    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.637711 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.637664    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b" podStartSLOduration=1.637644364 podStartE2EDuration="1.637644364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:40.63616579 +0000 UTC m=+943.110108776" watchObservedRunningTime="2026-09-17 23:21:40.637644364 +0000 UTC m=+943.111587349"
Sep 17 23:21:40.647122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.647098    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"]
Sep 17 23:21:40.663883 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.663843    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp" podStartSLOduration=1.6638306649999999 podStartE2EDuration="1.663830665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:40.663059144 +0000 UTC m=+943.137002128" watchObservedRunningTime="2026-09-17 23:21:40.663830665 +0000 UTC m=+943.137773648"
Sep 17 23:21:40.720036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.720010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2856ab-8c97-423a-8487-ec62b3989483-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.720131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.720116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6x8z\" (UniqueName: \"kubernetes.io/projected/237aa454-21cb-4841-9f6a-45bd31700e9c-kube-api-access-k6x8z\") pod \"payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.720200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.720157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrfm\" (UniqueName: \"kubernetes.io/projected/fe2856ab-8c97-423a-8487-ec62b3989483-kube-api-access-gxrfm\") pod \"payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.720281 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.720260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237aa454-21cb-4841-9f6a-45bd31700e9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.821771 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.821743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6x8z\" (UniqueName: \"kubernetes.io/projected/237aa454-21cb-4841-9f6a-45bd31700e9c-kube-api-access-k6x8z\") pod \"payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.821903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.821776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrfm\" (UniqueName: \"kubernetes.io/projected/fe2856ab-8c97-423a-8487-ec62b3989483-kube-api-access-gxrfm\") pod \"payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.821903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.821793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237aa454-21cb-4841-9f6a-45bd31700e9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.821903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.821869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2856ab-8c97-423a-8487-ec62b3989483-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.822715 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.822695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2856ab-8c97-423a-8487-ec62b3989483-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.822795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.822735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237aa454-21cb-4841-9f6a-45bd31700e9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.834750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.834711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6x8z\" (UniqueName: \"kubernetes.io/projected/237aa454-21cb-4841-9f6a-45bd31700e9c-kube-api-access-k6x8z\") pod \"payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.834934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.834914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrfm\" (UniqueName: \"kubernetes.io/projected/fe2856ab-8c97-423a-8487-ec62b3989483-kube-api-access-gxrfm\") pod \"payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:40.908097 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.908028    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:40.946923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:40.946887    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:41.200140 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.200115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"]
Sep 17 23:21:41.202104 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:41.202069    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237aa454_21cb_4841_9f6a_45bd31700e9c.slice/crio-ebec4c7e38be0bea3b941514b73108de88382b399d12b3734372895325500f71 WatchSource:0}: Error finding container ebec4c7e38be0bea3b941514b73108de88382b399d12b3734372895325500f71: Status 404 returned error can't find the container with id ebec4c7e38be0bea3b941514b73108de88382b399d12b3734372895325500f71
Sep 17 23:21:41.232432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.232409    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"]
Sep 17 23:21:41.239872 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:41.239847    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2856ab_8c97_423a_8487_ec62b3989483.slice/crio-e764b6cbbe758e082a5dcc1e45d8c01868447406e4efa1c42d6db7d5f0325c3c WatchSource:0}: Error finding container e764b6cbbe758e082a5dcc1e45d8c01868447406e4efa1c42d6db7d5f0325c3c: Status 404 returned error can't find the container with id e764b6cbbe758e082a5dcc1e45d8c01868447406e4efa1c42d6db7d5f0325c3c
Sep 17 23:21:41.621475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.621378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w" event={"ID":"fe2856ab-8c97-423a-8487-ec62b3989483","Type":"ContainerStarted","Data":"812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5"}
Sep 17 23:21:41.621475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.621441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w" event={"ID":"fe2856ab-8c97-423a-8487-ec62b3989483","Type":"ContainerStarted","Data":"e764b6cbbe758e082a5dcc1e45d8c01868447406e4efa1c42d6db7d5f0325c3c"}
Sep 17 23:21:41.621980 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.621489    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:41.623892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.623834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57" event={"ID":"237aa454-21cb-4841-9f6a-45bd31700e9c","Type":"ContainerStarted","Data":"79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb"}
Sep 17 23:21:41.624029 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.623923    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:41.624029 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.623962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57" event={"ID":"237aa454-21cb-4841-9f6a-45bd31700e9c","Type":"ContainerStarted","Data":"ebec4c7e38be0bea3b941514b73108de88382b399d12b3734372895325500f71"}
Sep 17 23:21:41.624384 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.624308    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7861-719527" podUID="6e43f6ab-8359-46a2-be8c-787253660640" containerName="test-curl-7861-719527" containerID="cri-o://27adbfc9522c2d728387ee811d244bf8c134c49e30bae6d57a9f59eb8c0b0e4b" gracePeriod=2
Sep 17 23:21:41.641077 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.641024    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w" podStartSLOduration=1.641010984 podStartE2EDuration="1.641010984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:41.638392745 +0000 UTC m=+944.112335747" watchObservedRunningTime="2026-09-17 23:21:41.641010984 +0000 UTC m=+944.114954002"
Sep 17 23:21:41.658324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.658280    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57" podStartSLOduration=1.658243406 podStartE2EDuration="1.658243406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:41.65735843 +0000 UTC m=+944.131301416" watchObservedRunningTime="2026-09-17 23:21:41.658243406 +0000 UTC m=+944.132186391"
Sep 17 23:21:41.660609 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:21:41.660571    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7861-719527\" is forbidden: User \"system:node:ip-10-0-143-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-250.ec2.internal' and this object" podUID="6e43f6ab-8359-46a2-be8c-787253660640" pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:41.683096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.683069    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"]
Sep 17 23:21:41.686489 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.686467    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.688676 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.688652    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0c363f\""
Sep 17 23:21:41.688824 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.688798    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0c363f-dockercfg-v78dp\""
Sep 17 23:21:41.699321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.699298    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"]
Sep 17 23:21:41.834211 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.834183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40446d60-31db-403b-a8a2-a1b3f908ac02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.834367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.834238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstqg\" (UniqueName: \"kubernetes.io/projected/40446d60-31db-403b-a8a2-a1b3f908ac02-kube-api-access-zstqg\") pod \"payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.935583 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.935519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstqg\" (UniqueName: \"kubernetes.io/projected/40446d60-31db-403b-a8a2-a1b3f908ac02-kube-api-access-zstqg\") pod \"payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.935696 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.935606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40446d60-31db-403b-a8a2-a1b3f908ac02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.936283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.936260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40446d60-31db-403b-a8a2-a1b3f908ac02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.947895 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.947873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstqg\" (UniqueName: \"kubernetes.io/projected/40446d60-31db-403b-a8a2-a1b3f908ac02-kube-api-access-zstqg\") pod \"payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:41.997120 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:41.997097    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:42.280565 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:42.280540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"]
Sep 17 23:21:42.282157 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:42.282132    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40446d60_31db_403b_a8a2_a1b3f908ac02.slice/crio-2effe9cebf3346fc3ad08cd73dde92e8c075719814fb38961af6c08a7a2c9c09 WatchSource:0}: Error finding container 2effe9cebf3346fc3ad08cd73dde92e8c075719814fb38961af6c08a7a2c9c09: Status 404 returned error can't find the container with id 2effe9cebf3346fc3ad08cd73dde92e8c075719814fb38961af6c08a7a2c9c09
Sep 17 23:21:42.643233 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:42.643137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd" event={"ID":"40446d60-31db-403b-a8a2-a1b3f908ac02","Type":"ContainerStarted","Data":"8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47"}
Sep 17 23:21:42.643671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:42.643239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd" event={"ID":"40446d60-31db-403b-a8a2-a1b3f908ac02","Type":"ContainerStarted","Data":"2effe9cebf3346fc3ad08cd73dde92e8c075719814fb38961af6c08a7a2c9c09"}
Sep 17 23:21:42.643671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:42.643632    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:42.659658 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:42.659606    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd" podStartSLOduration=1.6595897979999998 podStartE2EDuration="1.659589798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:42.65866807 +0000 UTC m=+945.132611054" watchObservedRunningTime="2026-09-17 23:21:42.659589798 +0000 UTC m=+945.133532783"
Sep 17 23:21:43.649101 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:43.649076    2559 generic.go:356] "Generic (PLEG): container finished" podID="6e43f6ab-8359-46a2-be8c-787253660640" containerID="27adbfc9522c2d728387ee811d244bf8c134c49e30bae6d57a9f59eb8c0b0e4b" exitCode=137
Sep 17 23:21:43.771013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:43.770986    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:43.954407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:43.954382    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqcsc\" (UniqueName: \"kubernetes.io/projected/6e43f6ab-8359-46a2-be8c-787253660640-kube-api-access-tqcsc\") pod \"6e43f6ab-8359-46a2-be8c-787253660640\" (UID: \"6e43f6ab-8359-46a2-be8c-787253660640\") "
Sep 17 23:21:43.956475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:43.956447    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e43f6ab-8359-46a2-be8c-787253660640-kube-api-access-tqcsc" (OuterVolumeSpecName: "kube-api-access-tqcsc") pod "6e43f6ab-8359-46a2-be8c-787253660640" (UID: "6e43f6ab-8359-46a2-be8c-787253660640"). InnerVolumeSpecName "kube-api-access-tqcsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:44.055215 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:44.055190    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqcsc\" (UniqueName: \"kubernetes.io/projected/6e43f6ab-8359-46a2-be8c-787253660640-kube-api-access-tqcsc\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:21:44.655965 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:44.655944    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:44.656310 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:44.655985    2559 scope.go:117] "RemoveContainer" containerID="27adbfc9522c2d728387ee811d244bf8c134c49e30bae6d57a9f59eb8c0b0e4b"
Sep 17 23:21:44.668340 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:21:44.668314    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7861-719527\" is forbidden: User \"system:node:ip-10-0-143-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-250.ec2.internal' and this object" podUID="6e43f6ab-8359-46a2-be8c-787253660640" pod="opendatahub/test-curl-7861-719527"
Sep 17 23:21:45.639442 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:45.639412    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e43f6ab-8359-46a2-be8c-787253660640" path="/var/lib/kubelet/pods/6e43f6ab-8359-46a2-be8c-787253660640/volumes"
Sep 17 23:21:51.626536 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:51.626495    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:21:51.626998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:51.626765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:21:52.645574 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:52.645548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:21:52.645975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:52.645631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:21:53.158900 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.158867    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"]
Sep 17 23:21:53.165649 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.165633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.168038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.168008    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-021e6520-dockercfg-szxd7\""
Sep 17 23:21:53.168038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.168023    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-021e6520\""
Sep 17 23:21:53.177599 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.177576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"]
Sep 17 23:21:53.267846 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.267820    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"]
Sep 17 23:21:53.271890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.271875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.284156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.284133    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"]
Sep 17 23:21:53.327142 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.327112    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c58adee-c4a3-4284-b649-6f87cc908853-plugins-config-volume\") pod \"payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") " pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.327282 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.327161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9flr\" (UniqueName: \"kubernetes.io/projected/6c58adee-c4a3-4284-b649-6f87cc908853-kube-api-access-p9flr\") pod \"payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") " pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.427930 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.427872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9flr\" (UniqueName: \"kubernetes.io/projected/6c58adee-c4a3-4284-b649-6f87cc908853-kube-api-access-p9flr\") pod \"payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") " pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.427930 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.427915    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qtv\" (UniqueName: \"kubernetes.io/projected/f5203f92-3ded-426c-87d7-c643dbf285ff-kube-api-access-p5qtv\") pod \"payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.428064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.427970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5203f92-3ded-426c-87d7-c643dbf285ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.428064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.428033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c58adee-c4a3-4284-b649-6f87cc908853-plugins-config-volume\") pod \"payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") " pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.428747 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.428706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c58adee-c4a3-4284-b649-6f87cc908853-plugins-config-volume\") pod \"payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") " pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.437049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.437025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9flr\" (UniqueName: \"kubernetes.io/projected/6c58adee-c4a3-4284-b649-6f87cc908853-kube-api-access-p9flr\") pod \"payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") " pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.475911 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.475886    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:53.529823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.529754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qtv\" (UniqueName: \"kubernetes.io/projected/f5203f92-3ded-426c-87d7-c643dbf285ff-kube-api-access-p5qtv\") pod \"payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.529942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.529841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5203f92-3ded-426c-87d7-c643dbf285ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.530566 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.530543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5203f92-3ded-426c-87d7-c643dbf285ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.546426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.546374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qtv\" (UniqueName: \"kubernetes.io/projected/f5203f92-3ded-426c-87d7-c643dbf285ff-kube-api-access-p5qtv\") pod \"payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.582666 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.582642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:53.651807 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.651779    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:21:53.756354 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.756330    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"]
Sep 17 23:21:53.758896 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:53.758863    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c58adee_c4a3_4284_b649_6f87cc908853.slice/crio-b652f2e3304ec0ff7105ebbe5daf3fb18f7d51e54879944d63b7f970894730b7 WatchSource:0}: Error finding container b652f2e3304ec0ff7105ebbe5daf3fb18f7d51e54879944d63b7f970894730b7: Status 404 returned error can't find the container with id b652f2e3304ec0ff7105ebbe5daf3fb18f7d51e54879944d63b7f970894730b7
Sep 17 23:21:53.858861 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:53.858835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"]
Sep 17 23:21:53.861485 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:53.861463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5203f92_3ded_426c_87d7_c643dbf285ff.slice/crio-1926b0d10d5ea196d21c595473e400e86017a52c57925c205c6c81fffbe96f2b WatchSource:0}: Error finding container 1926b0d10d5ea196d21c595473e400e86017a52c57925c205c6c81fffbe96f2b: Status 404 returned error can't find the container with id 1926b0d10d5ea196d21c595473e400e86017a52c57925c205c6c81fffbe96f2b
Sep 17 23:21:54.712397 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.712361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n" event={"ID":"f5203f92-3ded-426c-87d7-c643dbf285ff","Type":"ContainerStarted","Data":"07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4"}
Sep 17 23:21:54.712942 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.712903    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:21:54.713093 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.712955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n" event={"ID":"f5203f92-3ded-426c-87d7-c643dbf285ff","Type":"ContainerStarted","Data":"1926b0d10d5ea196d21c595473e400e86017a52c57925c205c6c81fffbe96f2b"}
Sep 17 23:21:54.714301 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.714274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm" event={"ID":"6c58adee-c4a3-4284-b649-6f87cc908853","Type":"ContainerStarted","Data":"477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82"}
Sep 17 23:21:54.714434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.714323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm" event={"ID":"6c58adee-c4a3-4284-b649-6f87cc908853","Type":"ContainerStarted","Data":"b652f2e3304ec0ff7105ebbe5daf3fb18f7d51e54879944d63b7f970894730b7"}
Sep 17 23:21:54.714434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.714370    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:21:54.731791 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.731736    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n" podStartSLOduration=1.731707895 podStartE2EDuration="1.731707895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:54.729614775 +0000 UTC m=+957.203557758" watchObservedRunningTime="2026-09-17 23:21:54.731707895 +0000 UTC m=+957.205650919"
Sep 17 23:21:54.747667 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:54.747626    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm" podStartSLOduration=1.7476134989999998 podStartE2EDuration="1.747613499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:54.745840963 +0000 UTC m=+957.219783946" watchObservedRunningTime="2026-09-17 23:21:54.747613499 +0000 UTC m=+957.221556481"
Sep 17 23:21:56.127085 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:56.127052    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:21:57.276517 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.276487    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"]
Sep 17 23:21:57.281921 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.281895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.286038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.286017    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-63b144\""
Sep 17 23:21:57.286137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.286056    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-63b144-dockercfg-fttdq\""
Sep 17 23:21:57.294425 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.294404    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"]
Sep 17 23:21:57.361401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.361377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgn4\" (UniqueName: \"kubernetes.io/projected/ad4df59f-f3b1-405e-954e-03b460a06bcc-kube-api-access-8wgn4\") pod \"payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") " pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.361489 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.361454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4df59f-f3b1-405e-954e-03b460a06bcc-plugins-config-volume\") pod \"payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") " pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.461946 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.461914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4df59f-f3b1-405e-954e-03b460a06bcc-plugins-config-volume\") pod \"payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") " pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.462030 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.461977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgn4\" (UniqueName: \"kubernetes.io/projected/ad4df59f-f3b1-405e-954e-03b460a06bcc-kube-api-access-8wgn4\") pod \"payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") " pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.462651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.462634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4df59f-f3b1-405e-954e-03b460a06bcc-plugins-config-volume\") pod \"payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") " pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.470141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.470116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgn4\" (UniqueName: \"kubernetes.io/projected/ad4df59f-f3b1-405e-954e-03b460a06bcc-kube-api-access-8wgn4\") pod \"payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") " pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.593133 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.593080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:57.865453 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:57.865425    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"]
Sep 17 23:21:57.867949 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:21:57.867919    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4df59f_f3b1_405e_954e_03b460a06bcc.slice/crio-79eb822992e19a9bcb5f0932c2de96cca96b98e310504a6fb1dd0234fdb44ca1 WatchSource:0}: Error finding container 79eb822992e19a9bcb5f0932c2de96cca96b98e310504a6fb1dd0234fdb44ca1: Status 404 returned error can't find the container with id 79eb822992e19a9bcb5f0932c2de96cca96b98e310504a6fb1dd0234fdb44ca1
Sep 17 23:21:58.737957 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:58.737912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4" event={"ID":"ad4df59f-f3b1-405e-954e-03b460a06bcc","Type":"ContainerStarted","Data":"5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f"}
Sep 17 23:21:58.738416 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:58.738013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4" event={"ID":"ad4df59f-f3b1-405e-954e-03b460a06bcc","Type":"ContainerStarted","Data":"79eb822992e19a9bcb5f0932c2de96cca96b98e310504a6fb1dd0234fdb44ca1"}
Sep 17 23:21:58.738416 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:58.738047    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:21:58.758051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:21:58.758003    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4" podStartSLOduration=1.7579852009999999 podStartE2EDuration="1.757985201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:58.757651808 +0000 UTC m=+961.231594804" watchObservedRunningTime="2026-09-17 23:21:58.757985201 +0000 UTC m=+961.231928188"
Sep 17 23:22:00.676461 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.676430    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"]
Sep 17 23:22:00.681254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.681236    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.683640 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.683617    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-307a7b-dockercfg-w9zr4\""
Sep 17 23:22:00.683820 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.683800    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-307a7b\""
Sep 17 23:22:00.689398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.689378    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"]
Sep 17 23:22:00.754581 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.754557    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"]
Sep 17 23:22:00.757953 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.757938    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.768858 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.768837    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"]
Sep 17 23:22:00.793175 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.793155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzkxl\" (UniqueName: \"kubernetes.io/projected/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-kube-api-access-wzkxl\") pod \"payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.793267 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.793183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p52v\" (UniqueName: \"kubernetes.io/projected/73895d71-0276-4b59-b282-0fd88c6747a3-kube-api-access-2p52v\") pod \"payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.793267 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.793206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73895d71-0276-4b59-b282-0fd88c6747a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.793344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.793305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.893898 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.893862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.893996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.893936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzkxl\" (UniqueName: \"kubernetes.io/projected/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-kube-api-access-wzkxl\") pod \"payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.893996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.893955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p52v\" (UniqueName: \"kubernetes.io/projected/73895d71-0276-4b59-b282-0fd88c6747a3-kube-api-access-2p52v\") pod \"payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.893996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.893976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73895d71-0276-4b59-b282-0fd88c6747a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.894681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.894661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73895d71-0276-4b59-b282-0fd88c6747a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.894681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.894674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.901688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.901668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p52v\" (UniqueName: \"kubernetes.io/projected/73895d71-0276-4b59-b282-0fd88c6747a3-kube-api-access-2p52v\") pod \"payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:00.901807 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.901790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzkxl\" (UniqueName: \"kubernetes.io/projected/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-kube-api-access-wzkxl\") pod \"payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:00.993444 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:00.993406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:01.067996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.067971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:01.265686 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.265661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"]
Sep 17 23:22:01.267422 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:22:01.267396    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbf0b6c_bdd6_462f_a89f_8b52155d25f2.slice/crio-b03f551422c86bf11175e190c88255841b17da17a820f7200c8bb7f090b2f936 WatchSource:0}: Error finding container b03f551422c86bf11175e190c88255841b17da17a820f7200c8bb7f090b2f936: Status 404 returned error can't find the container with id b03f551422c86bf11175e190c88255841b17da17a820f7200c8bb7f090b2f936
Sep 17 23:22:01.347914 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.347890    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"]
Sep 17 23:22:01.350286 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:22:01.350255    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73895d71_0276_4b59_b282_0fd88c6747a3.slice/crio-d08c752421eb4b5d67b63ed856d89efde64bf31888ba7402720d828ffd7a30d6 WatchSource:0}: Error finding container d08c752421eb4b5d67b63ed856d89efde64bf31888ba7402720d828ffd7a30d6: Status 404 returned error can't find the container with id d08c752421eb4b5d67b63ed856d89efde64bf31888ba7402720d828ffd7a30d6
Sep 17 23:22:01.764352 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.764317    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc" event={"ID":"73895d71-0276-4b59-b282-0fd88c6747a3","Type":"ContainerStarted","Data":"b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371"}
Sep 17 23:22:01.764825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.764405    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:01.764825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.764427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc" event={"ID":"73895d71-0276-4b59-b282-0fd88c6747a3","Type":"ContainerStarted","Data":"d08c752421eb4b5d67b63ed856d89efde64bf31888ba7402720d828ffd7a30d6"}
Sep 17 23:22:01.766017 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.765988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752" event={"ID":"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2","Type":"ContainerStarted","Data":"9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025"}
Sep 17 23:22:01.766141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.766041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:01.766141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.766063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752" event={"ID":"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2","Type":"ContainerStarted","Data":"b03f551422c86bf11175e190c88255841b17da17a820f7200c8bb7f090b2f936"}
Sep 17 23:22:01.781365 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.781326    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc" podStartSLOduration=1.781310435 podStartE2EDuration="1.781310435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:01.780243019 +0000 UTC m=+964.254186003" watchObservedRunningTime="2026-09-17 23:22:01.781310435 +0000 UTC m=+964.255253420"
Sep 17 23:22:01.797266 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:01.797207    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752" podStartSLOduration=1.7971897430000001 podStartE2EDuration="1.797189743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:01.796586044 +0000 UTC m=+964.270529026" watchObservedRunningTime="2026-09-17 23:22:01.797189743 +0000 UTC m=+964.271132727"
Sep 17 23:22:05.721458 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:05.721428    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:22:05.721929 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:05.721502    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:22:09.744872 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:09.744845    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:22:12.773681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:12.773606    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:22:12.773681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:12.773676    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:22:18.742773 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:18.742739    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:22:28.347060 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.347022    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"]
Sep 17 23:22:28.347528 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.347282    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm" podUID="6c58adee-c4a3-4284-b649-6f87cc908853" containerName="payload-processing" containerID="cri-o://477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82" gracePeriod=30
Sep 17 23:22:28.355452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.355428    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"]
Sep 17 23:22:28.355635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.355614    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n" podUID="f5203f92-3ded-426c-87d7-c643dbf285ff" containerName="payload-pre-processing" containerID="cri-o://07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4" gracePeriod=30
Sep 17 23:22:28.628113 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.628065    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:22:28.631207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.631179    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:22:28.722049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.722020    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5qtv\" (UniqueName: \"kubernetes.io/projected/f5203f92-3ded-426c-87d7-c643dbf285ff-kube-api-access-p5qtv\") pod \"f5203f92-3ded-426c-87d7-c643dbf285ff\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") "
Sep 17 23:22:28.722216 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.722059    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5203f92-3ded-426c-87d7-c643dbf285ff-plugins-config-volume\") pod \"f5203f92-3ded-426c-87d7-c643dbf285ff\" (UID: \"f5203f92-3ded-426c-87d7-c643dbf285ff\") "
Sep 17 23:22:28.722216 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.722133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c58adee-c4a3-4284-b649-6f87cc908853-plugins-config-volume\") pod \"6c58adee-c4a3-4284-b649-6f87cc908853\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") "
Sep 17 23:22:28.722216 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.722166    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9flr\" (UniqueName: \"kubernetes.io/projected/6c58adee-c4a3-4284-b649-6f87cc908853-kube-api-access-p9flr\") pod \"6c58adee-c4a3-4284-b649-6f87cc908853\" (UID: \"6c58adee-c4a3-4284-b649-6f87cc908853\") "
Sep 17 23:22:28.722479 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.722444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5203f92-3ded-426c-87d7-c643dbf285ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5203f92-3ded-426c-87d7-c643dbf285ff" (UID: "f5203f92-3ded-426c-87d7-c643dbf285ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:28.722574 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.722518    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c58adee-c4a3-4284-b649-6f87cc908853-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c58adee-c4a3-4284-b649-6f87cc908853" (UID: "6c58adee-c4a3-4284-b649-6f87cc908853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:28.724431 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.724407    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c58adee-c4a3-4284-b649-6f87cc908853-kube-api-access-p9flr" (OuterVolumeSpecName: "kube-api-access-p9flr") pod "6c58adee-c4a3-4284-b649-6f87cc908853" (UID: "6c58adee-c4a3-4284-b649-6f87cc908853"). InnerVolumeSpecName "kube-api-access-p9flr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:22:28.724527 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.724436    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5203f92-3ded-426c-87d7-c643dbf285ff-kube-api-access-p5qtv" (OuterVolumeSpecName: "kube-api-access-p5qtv") pod "f5203f92-3ded-426c-87d7-c643dbf285ff" (UID: "f5203f92-3ded-426c-87d7-c643dbf285ff"). InnerVolumeSpecName "kube-api-access-p5qtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:22:28.822999 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.822976    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qtv\" (UniqueName: \"kubernetes.io/projected/f5203f92-3ded-426c-87d7-c643dbf285ff-kube-api-access-p5qtv\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:22:28.822999 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.822999    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5203f92-3ded-426c-87d7-c643dbf285ff-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:22:28.823116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.823008    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c58adee-c4a3-4284-b649-6f87cc908853-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:22:28.823116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.823017    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9flr\" (UniqueName: \"kubernetes.io/projected/6c58adee-c4a3-4284-b649-6f87cc908853-kube-api-access-p9flr\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:22:28.907251 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.907198    2559 generic.go:356] "Generic (PLEG): container finished" podID="6c58adee-c4a3-4284-b649-6f87cc908853" containerID="477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82" exitCode=0
Sep 17 23:22:28.907251 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.907241    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"
Sep 17 23:22:28.907408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.907293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm" event={"ID":"6c58adee-c4a3-4284-b649-6f87cc908853","Type":"ContainerDied","Data":"477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82"}
Sep 17 23:22:28.907408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.907341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm" event={"ID":"6c58adee-c4a3-4284-b649-6f87cc908853","Type":"ContainerDied","Data":"b652f2e3304ec0ff7105ebbe5daf3fb18f7d51e54879944d63b7f970894730b7"}
Sep 17 23:22:28.907408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.907364    2559 scope.go:117] "RemoveContainer" containerID="477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82"
Sep 17 23:22:28.908837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.908813    2559 generic.go:356] "Generic (PLEG): container finished" podID="f5203f92-3ded-426c-87d7-c643dbf285ff" containerID="07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4" exitCode=0
Sep 17 23:22:28.908944 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.908850    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"
Sep 17 23:22:28.908944 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.908889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n" event={"ID":"f5203f92-3ded-426c-87d7-c643dbf285ff","Type":"ContainerDied","Data":"07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4"}
Sep 17 23:22:28.909058 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.908945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n" event={"ID":"f5203f92-3ded-426c-87d7-c643dbf285ff","Type":"ContainerDied","Data":"1926b0d10d5ea196d21c595473e400e86017a52c57925c205c6c81fffbe96f2b"}
Sep 17 23:22:28.917174 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.917156    2559 scope.go:117] "RemoveContainer" containerID="477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82"
Sep 17 23:22:28.917581 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:22:28.917419    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82\": container with ID starting with 477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82 not found: ID does not exist" containerID="477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82"
Sep 17 23:22:28.917581 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.917449    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82"} err="failed to get container status \"477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82\": rpc error: code = NotFound desc = could not find container \"477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82\": container with ID starting with 477676b29c9e74dfd4c28d049c238d1ce1272ee358eb7d6d18a7dbdf98defb82 not found: ID does not exist"
Sep 17 23:22:28.917581 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.917467    2559 scope.go:117] "RemoveContainer" containerID="07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4"
Sep 17 23:22:28.925374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.925354    2559 scope.go:117] "RemoveContainer" containerID="07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4"
Sep 17 23:22:28.925613 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:22:28.925598    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4\": container with ID starting with 07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4 not found: ID does not exist" containerID="07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4"
Sep 17 23:22:28.925659 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.925619    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4"} err="failed to get container status \"07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4\": rpc error: code = NotFound desc = could not find container \"07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4\": container with ID starting with 07b2a410341c4e7ad8415430d1ba45731249b1872346460345b8a39917daa3e4 not found: ID does not exist"
Sep 17 23:22:28.933662 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.933640    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"]
Sep 17 23:22:28.936111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.936088    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021e6520-55b5f8dfb5-2d8rm"]
Sep 17 23:22:28.947129 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.947107    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"]
Sep 17 23:22:28.950315 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:28.950295    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021e6520-84c45954fd-nrm4n"]
Sep 17 23:22:29.639271 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:29.639204    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c58adee-c4a3-4284-b649-6f87cc908853" path="/var/lib/kubelet/pods/6c58adee-c4a3-4284-b649-6f87cc908853/volumes"
Sep 17 23:22:29.639586 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:29.639547    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5203f92-3ded-426c-87d7-c643dbf285ff" path="/var/lib/kubelet/pods/f5203f92-3ded-426c-87d7-c643dbf285ff/volumes"
Sep 17 23:22:33.535964 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:33.535927    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:22:34.931028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.930994    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"]
Sep 17 23:22:34.931613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.931589    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5203f92-3ded-426c-87d7-c643dbf285ff" containerName="payload-pre-processing"
Sep 17 23:22:34.931694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.931615    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5203f92-3ded-426c-87d7-c643dbf285ff" containerName="payload-pre-processing"
Sep 17 23:22:34.931694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.931635    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c58adee-c4a3-4284-b649-6f87cc908853" containerName="payload-processing"
Sep 17 23:22:34.931694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.931644    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c58adee-c4a3-4284-b649-6f87cc908853" containerName="payload-processing"
Sep 17 23:22:34.931910 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.931755    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c58adee-c4a3-4284-b649-6f87cc908853" containerName="payload-processing"
Sep 17 23:22:34.931910 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.931774    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5203f92-3ded-426c-87d7-c643dbf285ff" containerName="payload-pre-processing"
Sep 17 23:22:34.936599 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.936575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:34.938804 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.938783    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d683cd6a\""
Sep 17 23:22:34.938937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.938864    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d683cd6a-dockercfg-spzlq\""
Sep 17 23:22:34.947545 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:34.947518    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"]
Sep 17 23:22:35.070351 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.070325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6311995-ce75-49f9-b4b7-84f8430eb211-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.070464 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.070365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4grg\" (UniqueName: \"kubernetes.io/projected/e6311995-ce75-49f9-b4b7-84f8430eb211-kube-api-access-r4grg\") pod \"payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.079211 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.079187    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"]
Sep 17 23:22:35.082712 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.082696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.094204 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.094184    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"]
Sep 17 23:22:35.171150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.171127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnft\" (UniqueName: \"kubernetes.io/projected/ce7988fb-5a22-4226-8fb4-234d3450d744-kube-api-access-7vnft\") pod \"payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") " pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.171250 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.171186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6311995-ce75-49f9-b4b7-84f8430eb211-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.171333 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.171313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7988fb-5a22-4226-8fb4-234d3450d744-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") " pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.171372 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.171350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4grg\" (UniqueName: \"kubernetes.io/projected/e6311995-ce75-49f9-b4b7-84f8430eb211-kube-api-access-r4grg\") pod \"payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.171912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.171896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6311995-ce75-49f9-b4b7-84f8430eb211-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.179304 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.179287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4grg\" (UniqueName: \"kubernetes.io/projected/e6311995-ce75-49f9-b4b7-84f8430eb211-kube-api-access-r4grg\") pod \"payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.249689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.249668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.277795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.272994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7988fb-5a22-4226-8fb4-234d3450d744-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") " pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.277795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.273189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnft\" (UniqueName: \"kubernetes.io/projected/ce7988fb-5a22-4226-8fb4-234d3450d744-kube-api-access-7vnft\") pod \"payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") " pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.277795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.273803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7988fb-5a22-4226-8fb4-234d3450d744-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") " pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.283313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.283289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnft\" (UniqueName: \"kubernetes.io/projected/ce7988fb-5a22-4226-8fb4-234d3450d744-kube-api-access-7vnft\") pod \"payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") " pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.392297 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.392270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.523177 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.523148    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"]
Sep 17 23:22:35.524624 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:22:35.524601    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6311995_ce75_49f9_b4b7_84f8430eb211.slice/crio-5151d93ac01785c2f8c94e4fa719f631e76c74aece5516ed3d5b1995e0ba2254 WatchSource:0}: Error finding container 5151d93ac01785c2f8c94e4fa719f631e76c74aece5516ed3d5b1995e0ba2254: Status 404 returned error can't find the container with id 5151d93ac01785c2f8c94e4fa719f631e76c74aece5516ed3d5b1995e0ba2254
Sep 17 23:22:35.663133 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.663108    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"]
Sep 17 23:22:35.665098 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:22:35.665074    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7988fb_5a22_4226_8fb4_234d3450d744.slice/crio-1922b29eba72df3585d54ad496ee0e02ceb42c59f0f02f3d472e5946d0aafb7e WatchSource:0}: Error finding container 1922b29eba72df3585d54ad496ee0e02ceb42c59f0f02f3d472e5946d0aafb7e: Status 404 returned error can't find the container with id 1922b29eba72df3585d54ad496ee0e02ceb42c59f0f02f3d472e5946d0aafb7e
Sep 17 23:22:35.951361 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.951316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5" event={"ID":"ce7988fb-5a22-4226-8fb4-234d3450d744","Type":"ContainerStarted","Data":"a6e064dc65a3bac040f00a30aa6b2cf30f2b15c0bfa10c8db20615d5d5c504c8"}
Sep 17 23:22:35.951811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.951397    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:35.951811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.951422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5" event={"ID":"ce7988fb-5a22-4226-8fb4-234d3450d744","Type":"ContainerStarted","Data":"1922b29eba72df3585d54ad496ee0e02ceb42c59f0f02f3d472e5946d0aafb7e"}
Sep 17 23:22:35.953021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.952997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p" event={"ID":"e6311995-ce75-49f9-b4b7-84f8430eb211","Type":"ContainerStarted","Data":"ef10058aae50205e6888cc53b39f9db9c0906fa7c90d513bc3e7e3182105a15d"}
Sep 17 23:22:35.953118 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.953041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:35.953118 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.953056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p" event={"ID":"e6311995-ce75-49f9-b4b7-84f8430eb211","Type":"ContainerStarted","Data":"5151d93ac01785c2f8c94e4fa719f631e76c74aece5516ed3d5b1995e0ba2254"}
Sep 17 23:22:35.968333 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.968296    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5" podStartSLOduration=0.968284138 podStartE2EDuration="968.284138ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:35.966259523 +0000 UTC m=+998.440202503" watchObservedRunningTime="2026-09-17 23:22:35.968284138 +0000 UTC m=+998.442227111"
Sep 17 23:22:35.982818 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:35.982781    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p" podStartSLOduration=1.982767941 podStartE2EDuration="1.982767941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:35.981239518 +0000 UTC m=+998.455182504" watchObservedRunningTime="2026-09-17 23:22:35.982767941 +0000 UTC m=+998.456710926"
Sep 17 23:22:46.961824 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:46.961792    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:22:46.962344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:46.961866    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:22:49.533209 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:22:49.533167    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:23:03.936653 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:03.936618    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"]
Sep 17 23:23:03.937149 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:03.936898    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5" podUID="ce7988fb-5a22-4226-8fb4-234d3450d744" containerName="payload-processing" containerID="cri-o://a6e064dc65a3bac040f00a30aa6b2cf30f2b15c0bfa10c8db20615d5d5c504c8" gracePeriod=30
Sep 17 23:23:03.946669 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:03.946643    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"]
Sep 17 23:23:03.946905 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:03.946855    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p" podUID="e6311995-ce75-49f9-b4b7-84f8430eb211" containerName="payload-pre-processing" containerID="cri-o://ef10058aae50205e6888cc53b39f9db9c0906fa7c90d513bc3e7e3182105a15d" gracePeriod=30
Sep 17 23:23:04.115949 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.115836    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce7988fb-5a22-4226-8fb4-234d3450d744" containerID="a6e064dc65a3bac040f00a30aa6b2cf30f2b15c0bfa10c8db20615d5d5c504c8" exitCode=0
Sep 17 23:23:04.116150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.115932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5" event={"ID":"ce7988fb-5a22-4226-8fb4-234d3450d744","Type":"ContainerDied","Data":"a6e064dc65a3bac040f00a30aa6b2cf30f2b15c0bfa10c8db20615d5d5c504c8"}
Sep 17 23:23:04.118463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.118423    2559 generic.go:356] "Generic (PLEG): container finished" podID="e6311995-ce75-49f9-b4b7-84f8430eb211" containerID="ef10058aae50205e6888cc53b39f9db9c0906fa7c90d513bc3e7e3182105a15d" exitCode=0
Sep 17 23:23:04.118625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.118555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p" event={"ID":"e6311995-ce75-49f9-b4b7-84f8430eb211","Type":"ContainerDied","Data":"ef10058aae50205e6888cc53b39f9db9c0906fa7c90d513bc3e7e3182105a15d"}
Sep 17 23:23:04.198226 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.198175    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:23:04.230533 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.230508    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:23:04.310866 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.310835    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7988fb-5a22-4226-8fb4-234d3450d744-plugins-config-volume\") pod \"ce7988fb-5a22-4226-8fb4-234d3450d744\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") "
Sep 17 23:23:04.310994 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.310896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6311995-ce75-49f9-b4b7-84f8430eb211-plugins-config-volume\") pod \"e6311995-ce75-49f9-b4b7-84f8430eb211\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") "
Sep 17 23:23:04.310994 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.310915    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vnft\" (UniqueName: \"kubernetes.io/projected/ce7988fb-5a22-4226-8fb4-234d3450d744-kube-api-access-7vnft\") pod \"ce7988fb-5a22-4226-8fb4-234d3450d744\" (UID: \"ce7988fb-5a22-4226-8fb4-234d3450d744\") "
Sep 17 23:23:04.310994 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.310987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4grg\" (UniqueName: \"kubernetes.io/projected/e6311995-ce75-49f9-b4b7-84f8430eb211-kube-api-access-r4grg\") pod \"e6311995-ce75-49f9-b4b7-84f8430eb211\" (UID: \"e6311995-ce75-49f9-b4b7-84f8430eb211\") "
Sep 17 23:23:04.311265 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.311238    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7988fb-5a22-4226-8fb4-234d3450d744-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce7988fb-5a22-4226-8fb4-234d3450d744" (UID: "ce7988fb-5a22-4226-8fb4-234d3450d744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:04.311387 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.311366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6311995-ce75-49f9-b4b7-84f8430eb211-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6311995-ce75-49f9-b4b7-84f8430eb211" (UID: "e6311995-ce75-49f9-b4b7-84f8430eb211"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:04.312906 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.312885    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7988fb-5a22-4226-8fb4-234d3450d744-kube-api-access-7vnft" (OuterVolumeSpecName: "kube-api-access-7vnft") pod "ce7988fb-5a22-4226-8fb4-234d3450d744" (UID: "ce7988fb-5a22-4226-8fb4-234d3450d744"). InnerVolumeSpecName "kube-api-access-7vnft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:04.312984 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.312967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6311995-ce75-49f9-b4b7-84f8430eb211-kube-api-access-r4grg" (OuterVolumeSpecName: "kube-api-access-r4grg") pod "e6311995-ce75-49f9-b4b7-84f8430eb211" (UID: "e6311995-ce75-49f9-b4b7-84f8430eb211"). InnerVolumeSpecName "kube-api-access-r4grg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:04.412603 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.412578    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4grg\" (UniqueName: \"kubernetes.io/projected/e6311995-ce75-49f9-b4b7-84f8430eb211-kube-api-access-r4grg\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:23:04.412603 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.412599    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7988fb-5a22-4226-8fb4-234d3450d744-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:23:04.412793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.412611    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6311995-ce75-49f9-b4b7-84f8430eb211-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:23:04.412793 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:04.412624    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnft\" (UniqueName: \"kubernetes.io/projected/ce7988fb-5a22-4226-8fb4-234d3450d744-kube-api-access-7vnft\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:23:05.125754 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.125712    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"
Sep 17 23:23:05.125754 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.125712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5" event={"ID":"ce7988fb-5a22-4226-8fb4-234d3450d744","Type":"ContainerDied","Data":"1922b29eba72df3585d54ad496ee0e02ceb42c59f0f02f3d472e5946d0aafb7e"}
Sep 17 23:23:05.126356 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.125803    2559 scope.go:117] "RemoveContainer" containerID="a6e064dc65a3bac040f00a30aa6b2cf30f2b15c0bfa10c8db20615d5d5c504c8"
Sep 17 23:23:05.127318 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.127233    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"
Sep 17 23:23:05.127318 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.127277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p" event={"ID":"e6311995-ce75-49f9-b4b7-84f8430eb211","Type":"ContainerDied","Data":"5151d93ac01785c2f8c94e4fa719f631e76c74aece5516ed3d5b1995e0ba2254"}
Sep 17 23:23:05.136985 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.136962    2559 scope.go:117] "RemoveContainer" containerID="ef10058aae50205e6888cc53b39f9db9c0906fa7c90d513bc3e7e3182105a15d"
Sep 17 23:23:05.149115 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.149095    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"]
Sep 17 23:23:05.153436 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.153415    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d683cd6a-75687fbf7d-s7tx5"]
Sep 17 23:23:05.167993 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.167973    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"]
Sep 17 23:23:05.172954 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.172933    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d683cd6a-855d6b766b-nsz6p"]
Sep 17 23:23:05.640015 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.639988    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce7988fb-5a22-4226-8fb4-234d3450d744" path="/var/lib/kubelet/pods/ce7988fb-5a22-4226-8fb4-234d3450d744/volumes"
Sep 17 23:23:05.640349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:05.640335    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6311995-ce75-49f9-b4b7-84f8430eb211" path="/var/lib/kubelet/pods/e6311995-ce75-49f9-b4b7-84f8430eb211/volumes"
Sep 17 23:23:09.834113 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:09.834078    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:23:10.122892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.122814    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"]
Sep 17 23:23:10.123233 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.123219    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6311995-ce75-49f9-b4b7-84f8430eb211" containerName="payload-pre-processing"
Sep 17 23:23:10.123286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.123235    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6311995-ce75-49f9-b4b7-84f8430eb211" containerName="payload-pre-processing"
Sep 17 23:23:10.123286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.123259    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7988fb-5a22-4226-8fb4-234d3450d744" containerName="payload-processing"
Sep 17 23:23:10.123286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.123264    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7988fb-5a22-4226-8fb4-234d3450d744" containerName="payload-processing"
Sep 17 23:23:10.123407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.123338    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6311995-ce75-49f9-b4b7-84f8430eb211" containerName="payload-pre-processing"
Sep 17 23:23:10.123407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.123349    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7988fb-5a22-4226-8fb4-234d3450d744" containerName="payload-processing"
Sep 17 23:23:10.127822 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.127792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.130464 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.130434    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cc9b429c\""
Sep 17 23:23:10.130614 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.130526    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cc9b429c-dockercfg-dgcr8\""
Sep 17 23:23:10.138530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.138506    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"]
Sep 17 23:23:10.159328 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.159305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a9381-d3c7-4c68-9da2-cea5bfc512d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.159451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.159359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfsh\" (UniqueName: \"kubernetes.io/projected/692a9381-d3c7-4c68-9da2-cea5bfc512d4-kube-api-access-5qfsh\") pod \"payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.260459 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.260428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfsh\" (UniqueName: \"kubernetes.io/projected/692a9381-d3c7-4c68-9da2-cea5bfc512d4-kube-api-access-5qfsh\") pod \"payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.260573 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.260489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a9381-d3c7-4c68-9da2-cea5bfc512d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.261168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.261151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a9381-d3c7-4c68-9da2-cea5bfc512d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.268707 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.268687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfsh\" (UniqueName: \"kubernetes.io/projected/692a9381-d3c7-4c68-9da2-cea5bfc512d4-kube-api-access-5qfsh\") pod \"payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.439690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.439633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:10.713037 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:10.713003    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"]
Sep 17 23:23:10.717144 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:23:10.717117    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692a9381_d3c7_4c68_9da2_cea5bfc512d4.slice/crio-21a20d3eafea79ad745fc3644416face9de8412ff324babb54efa66728d6672c WatchSource:0}: Error finding container 21a20d3eafea79ad745fc3644416face9de8412ff324babb54efa66728d6672c: Status 404 returned error can't find the container with id 21a20d3eafea79ad745fc3644416face9de8412ff324babb54efa66728d6672c
Sep 17 23:23:11.162506 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:11.162421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv" event={"ID":"692a9381-d3c7-4c68-9da2-cea5bfc512d4","Type":"ContainerStarted","Data":"bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f"}
Sep 17 23:23:11.162506 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:11.162479    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv" event={"ID":"692a9381-d3c7-4c68-9da2-cea5bfc512d4","Type":"ContainerStarted","Data":"21a20d3eafea79ad745fc3644416face9de8412ff324babb54efa66728d6672c"}
Sep 17 23:23:11.162910 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:11.162568    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:11.178869 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:11.178830    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv" podStartSLOduration=1.178817409 podStartE2EDuration="1.178817409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:11.177430922 +0000 UTC m=+1033.651373904" watchObservedRunningTime="2026-09-17 23:23:11.178817409 +0000 UTC m=+1033.652760392"
Sep 17 23:23:22.171330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:22.171290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:24.634100 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:24.634057    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:23:37.930374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.930337    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"]
Sep 17 23:23:37.933865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.933850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:37.936977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.936951    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-861522\"/\"simulator-w0-861522-kserve-self-signed-certs\""
Sep 17 23:23:37.937117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.936951    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-861522\"/\"kube-root-ca.crt\""
Sep 17 23:23:37.937117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.936955    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-861522\"/\"openshift-service-ca.crt\""
Sep 17 23:23:37.937117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.936955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-861522\"/\"default-dockercfg-grqjc\""
Sep 17 23:23:37.943670 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:37.943648    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"]
Sep 17 23:23:38.085049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.085023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2z48\" (UniqueName: \"kubernetes.io/projected/31cc1d20-7e27-4b7e-9184-e47ad4539b57-kube-api-access-z2z48\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.085173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.085064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tmp-dir\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.085173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.085089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-model-cache\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.085173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.085139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-home\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.085173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.085168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-dshm\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.085313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.085212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tls-certs\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186447 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tls-certs\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2z48\" (UniqueName: \"kubernetes.io/projected/31cc1d20-7e27-4b7e-9184-e47ad4539b57-kube-api-access-z2z48\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tmp-dir\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-model-cache\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186715 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186577    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-home\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186807 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-dshm\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.186976 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.186949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tmp-dir\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.187094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.187052    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-model-cache\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.187094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.187064    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-home\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.188931 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.188902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-dshm\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.189025 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.189000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tls-certs\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.194032 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.194007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2z48\" (UniqueName: \"kubernetes.io/projected/31cc1d20-7e27-4b7e-9184-e47ad4539b57-kube-api-access-z2z48\") pod \"simulator-w0-861522-kserve-c4ffb5f4f-l62vz\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") " pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.243002 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.242977    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:38.512554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:38.512531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"]
Sep 17 23:23:38.515104 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:23:38.515078    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31cc1d20_7e27_4b7e_9184_e47ad4539b57.slice/crio-58a844562cc30d10e2d93f4e5aae5bdaf8dcb00e1707c11beddcaa547a386918 WatchSource:0}: Error finding container 58a844562cc30d10e2d93f4e5aae5bdaf8dcb00e1707c11beddcaa547a386918: Status 404 returned error can't find the container with id 58a844562cc30d10e2d93f4e5aae5bdaf8dcb00e1707c11beddcaa547a386918
Sep 17 23:23:39.298171 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.298140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"]
Sep 17 23:23:39.298649 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.298351    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv" podUID="692a9381-d3c7-4c68-9da2-cea5bfc512d4" containerName="payload-pre-processing" containerID="cri-o://bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f" gracePeriod=30
Sep 17 23:23:39.338513 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.338480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz" event={"ID":"31cc1d20-7e27-4b7e-9184-e47ad4539b57","Type":"ContainerStarted","Data":"9722f78b0623b3450aba3feb1d1dd6fde05fffff909169c220f5628ac8b37762"}
Sep 17 23:23:39.338622 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.338521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz" event={"ID":"31cc1d20-7e27-4b7e-9184-e47ad4539b57","Type":"ContainerStarted","Data":"58a844562cc30d10e2d93f4e5aae5bdaf8dcb00e1707c11beddcaa547a386918"}
Sep 17 23:23:39.357855 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.357799    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz" podStartSLOduration=2.35777703 podStartE2EDuration="2.35777703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:39.356418883 +0000 UTC m=+1061.830361866" watchObservedRunningTime="2026-09-17 23:23:39.35777703 +0000 UTC m=+1061.831720014"
Sep 17 23:23:39.552121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.552069    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:39.702388 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.702359    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qfsh\" (UniqueName: \"kubernetes.io/projected/692a9381-d3c7-4c68-9da2-cea5bfc512d4-kube-api-access-5qfsh\") pod \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") "
Sep 17 23:23:39.702561 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.702396    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a9381-d3c7-4c68-9da2-cea5bfc512d4-plugins-config-volume\") pod \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\" (UID: \"692a9381-d3c7-4c68-9da2-cea5bfc512d4\") "
Sep 17 23:23:39.702819 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.702790    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692a9381-d3c7-4c68-9da2-cea5bfc512d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "692a9381-d3c7-4c68-9da2-cea5bfc512d4" (UID: "692a9381-d3c7-4c68-9da2-cea5bfc512d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:39.704319 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.704289    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692a9381-d3c7-4c68-9da2-cea5bfc512d4-kube-api-access-5qfsh" (OuterVolumeSpecName: "kube-api-access-5qfsh") pod "692a9381-d3c7-4c68-9da2-cea5bfc512d4" (UID: "692a9381-d3c7-4c68-9da2-cea5bfc512d4"). InnerVolumeSpecName "kube-api-access-5qfsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:39.803501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.803437    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qfsh\" (UniqueName: \"kubernetes.io/projected/692a9381-d3c7-4c68-9da2-cea5bfc512d4-kube-api-access-5qfsh\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:23:39.803501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:39.803467    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a9381-d3c7-4c68-9da2-cea5bfc512d4-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:23:40.345628 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.345591    2559 generic.go:356] "Generic (PLEG): container finished" podID="692a9381-d3c7-4c68-9da2-cea5bfc512d4" containerID="bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f" exitCode=0
Sep 17 23:23:40.346057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.345643    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"
Sep 17 23:23:40.346057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.345673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv" event={"ID":"692a9381-d3c7-4c68-9da2-cea5bfc512d4","Type":"ContainerDied","Data":"bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f"}
Sep 17 23:23:40.346057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.345757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv" event={"ID":"692a9381-d3c7-4c68-9da2-cea5bfc512d4","Type":"ContainerDied","Data":"21a20d3eafea79ad745fc3644416face9de8412ff324babb54efa66728d6672c"}
Sep 17 23:23:40.346057 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.345783    2559 scope.go:117] "RemoveContainer" containerID="bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f"
Sep 17 23:23:40.354753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.354713    2559 scope.go:117] "RemoveContainer" containerID="bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f"
Sep 17 23:23:40.355022 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:23:40.355003    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f\": container with ID starting with bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f not found: ID does not exist" containerID="bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f"
Sep 17 23:23:40.355107 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.355028    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f"} err="failed to get container status \"bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f\": rpc error: code = NotFound desc = could not find container \"bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f\": container with ID starting with bf18c9d793306f76da19c73ebb3095d4e7ceb46f639efe82abf82ec4e0f0895f not found: ID does not exist"
Sep 17 23:23:40.366507 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.366485    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"]
Sep 17 23:23:40.368363 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:40.368342    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc9b429c-78c9d74b46-fhtkv"]
Sep 17 23:23:41.641422 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:41.641386    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692a9381-d3c7-4c68-9da2-cea5bfc512d4" path="/var/lib/kubelet/pods/692a9381-d3c7-4c68-9da2-cea5bfc512d4/volumes"
Sep 17 23:23:44.340290 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:44.340201    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:23:48.243135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:48.243088    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:48.243601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:48.243183    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:48.255776 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:48.255751    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:48.402973 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:48.402952    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:23:49.219500 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.219467    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"]
Sep 17 23:23:49.219902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.219889    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692a9381-d3c7-4c68-9da2-cea5bfc512d4" containerName="payload-pre-processing"
Sep 17 23:23:49.219960 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.219904    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="692a9381-d3c7-4c68-9da2-cea5bfc512d4" containerName="payload-pre-processing"
Sep 17 23:23:49.219996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.219962    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="692a9381-d3c7-4c68-9da2-cea5bfc512d4" containerName="payload-pre-processing"
Sep 17 23:23:49.223168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.223151    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.225891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.225869    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4f19d32d-dockercfg-b7rqx\""
Sep 17 23:23:49.226013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.225869    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4f19d32d\""
Sep 17 23:23:49.234213 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.234191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"]
Sep 17 23:23:49.254988 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.254954    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"]
Sep 17 23:23:49.258328 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.258311    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.266515 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.266491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"]
Sep 17 23:23:49.388044 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.388019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkr4\" (UniqueName: \"kubernetes.io/projected/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-kube-api-access-nfkr4\") pod \"payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.388194 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.388048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6c9k\" (UniqueName: \"kubernetes.io/projected/72374223-bb1b-4cbc-bc1a-5b2debcee787-kube-api-access-h6c9k\") pod \"payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") " pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.388194 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.388083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72374223-bb1b-4cbc-bc1a-5b2debcee787-plugins-config-volume\") pod \"payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") " pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.388290 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.388213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.489570 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.489507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.489570 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.489548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkr4\" (UniqueName: \"kubernetes.io/projected/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-kube-api-access-nfkr4\") pod \"payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.489570 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.489570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6c9k\" (UniqueName: \"kubernetes.io/projected/72374223-bb1b-4cbc-bc1a-5b2debcee787-kube-api-access-h6c9k\") pod \"payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") " pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.489848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.489601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72374223-bb1b-4cbc-bc1a-5b2debcee787-plugins-config-volume\") pod \"payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") " pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.490330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.490306    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72374223-bb1b-4cbc-bc1a-5b2debcee787-plugins-config-volume\") pod \"payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") " pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.490330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.490317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.497785 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.497751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkr4\" (UniqueName: \"kubernetes.io/projected/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-kube-api-access-nfkr4\") pod \"payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.498087 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.498067    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6c9k\" (UniqueName: \"kubernetes.io/projected/72374223-bb1b-4cbc-bc1a-5b2debcee787-kube-api-access-h6c9k\") pod \"payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") " pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.533493 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.533454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:49.569522 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.569494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:49.806529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.806458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"]
Sep 17 23:23:49.810354 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:23:49.810328    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3119a45_b6ed_4259_971f_7b2cfeb0fb60.slice/crio-d4762d28e1c1188eedf3dfe5e0961f79eae0b423016e4c1eff48d6b06b902c64 WatchSource:0}: Error finding container d4762d28e1c1188eedf3dfe5e0961f79eae0b423016e4c1eff48d6b06b902c64: Status 404 returned error can't find the container with id d4762d28e1c1188eedf3dfe5e0961f79eae0b423016e4c1eff48d6b06b902c64
Sep 17 23:23:49.847198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:49.847174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"]
Sep 17 23:23:49.850129 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:23:49.850103    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72374223_bb1b_4cbc_bc1a_5b2debcee787.slice/crio-3a80613c0e80e57de609d3d921c7e5d8d871b7fe8c48709c4b6f3e40d62986f3 WatchSource:0}: Error finding container 3a80613c0e80e57de609d3d921c7e5d8d871b7fe8c48709c4b6f3e40d62986f3: Status 404 returned error can't find the container with id 3a80613c0e80e57de609d3d921c7e5d8d871b7fe8c48709c4b6f3e40d62986f3
Sep 17 23:23:50.402976 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.402942    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv" event={"ID":"72374223-bb1b-4cbc-bc1a-5b2debcee787","Type":"ContainerStarted","Data":"15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7"}
Sep 17 23:23:50.403398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.402992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv" event={"ID":"72374223-bb1b-4cbc-bc1a-5b2debcee787","Type":"ContainerStarted","Data":"3a80613c0e80e57de609d3d921c7e5d8d871b7fe8c48709c4b6f3e40d62986f3"}
Sep 17 23:23:50.403398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.403055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:23:50.404689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.404665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f" event={"ID":"c3119a45-b6ed-4259-971f-7b2cfeb0fb60","Type":"ContainerStarted","Data":"08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b"}
Sep 17 23:23:50.404818 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.404709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f" event={"ID":"c3119a45-b6ed-4259-971f-7b2cfeb0fb60","Type":"ContainerStarted","Data":"d4762d28e1c1188eedf3dfe5e0961f79eae0b423016e4c1eff48d6b06b902c64"}
Sep 17 23:23:50.404818 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.404768    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:23:50.421137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.421089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv" podStartSLOduration=1.42107208 podStartE2EDuration="1.42107208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:50.418434937 +0000 UTC m=+1072.892377920" watchObservedRunningTime="2026-09-17 23:23:50.42107208 +0000 UTC m=+1072.895015064"
Sep 17 23:23:50.433890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:23:50.433854    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f" podStartSLOduration=1.4338407100000001 podStartE2EDuration="1.43384071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:50.432219882 +0000 UTC m=+1072.906162869" watchObservedRunningTime="2026-09-17 23:23:50.43384071 +0000 UTC m=+1072.907783692"
Sep 17 23:24:01.412305 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:01.412274    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:24:01.412796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:01.412351    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:24:02.337583 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:02.337535    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:24:19.846050 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:19.846010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:24:20.422123 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.422083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"]
Sep 17 23:24:20.426840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.426813    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.430041 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.430018    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-63b144\"/\"kube-root-ca.crt\""
Sep 17 23:24:20.430192 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.430034    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-63b144\"/\"default-dockercfg-gvdv9\""
Sep 17 23:24:20.430192 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.430044    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-63b144\"/\"autores-model-63b144-kserve-self-signed-certs\""
Sep 17 23:24:20.430388 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.430371    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-63b144\"/\"openshift-service-ca.crt\""
Sep 17 23:24:20.433452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.433429    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"]
Sep 17 23:24:20.552911 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.552879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65vk\" (UniqueName: \"kubernetes.io/projected/39b5bb69-edce-497a-81d1-9e1ab8821a56-kube-api-access-c65vk\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.553064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.552917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-tmp-dir\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.553064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.552978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-home\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.553064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.553015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-dshm\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.553203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.553092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b5bb69-edce-497a-81d1-9e1ab8821a56-tls-certs\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.553203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.553133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-model-cache\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.653666 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.653637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-model-cache\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.653803 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.653693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65vk\" (UniqueName: \"kubernetes.io/projected/39b5bb69-edce-497a-81d1-9e1ab8821a56-kube-api-access-c65vk\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.653803 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.653756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-tmp-dir\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.653953 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.653929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-home\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.654010 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.653976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-dshm\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.654103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.654084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b5bb69-edce-497a-81d1-9e1ab8821a56-tls-certs\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.654164 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.654114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-model-cache\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.654164 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.654142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-tmp-dir\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.654242 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.654223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-home\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.656133 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.656110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-dshm\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.656429 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.656413    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b5bb69-edce-497a-81d1-9e1ab8821a56-tls-certs\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.661267 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.661243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65vk\" (UniqueName: \"kubernetes.io/projected/39b5bb69-edce-497a-81d1-9e1ab8821a56-kube-api-access-c65vk\") pod \"autores-model-63b144-kserve-55c9c966f5-nsvz2\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") " pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:20.737246 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:20.737223    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:21.009625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:21.009573    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"]
Sep 17 23:24:21.012270 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:24:21.012237    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b5bb69_edce_497a_81d1_9e1ab8821a56.slice/crio-b4e7a4eb5f7b8c46455ac5f73fc9fbdd3dfab6f3c02eea206671874d1e5fb628 WatchSource:0}: Error finding container b4e7a4eb5f7b8c46455ac5f73fc9fbdd3dfab6f3c02eea206671874d1e5fb628: Status 404 returned error can't find the container with id b4e7a4eb5f7b8c46455ac5f73fc9fbdd3dfab6f3c02eea206671874d1e5fb628
Sep 17 23:24:21.581261 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:21.581218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2" event={"ID":"39b5bb69-edce-497a-81d1-9e1ab8821a56","Type":"ContainerStarted","Data":"30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad"}
Sep 17 23:24:21.581472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:21.581278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2" event={"ID":"39b5bb69-edce-497a-81d1-9e1ab8821a56","Type":"ContainerStarted","Data":"b4e7a4eb5f7b8c46455ac5f73fc9fbdd3dfab6f3c02eea206671874d1e5fb628"}
Sep 17 23:24:21.597842 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:21.597783    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2" podStartSLOduration=1.597763077 podStartE2EDuration="1.597763077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:21.596639595 +0000 UTC m=+1104.070582579" watchObservedRunningTime="2026-09-17 23:24:21.597763077 +0000 UTC m=+1104.071706059"
Sep 17 23:24:25.898356 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:25.898317    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"]
Sep 17 23:24:25.898890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:25.898570    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv" podUID="72374223-bb1b-4cbc-bc1a-5b2debcee787" containerName="payload-processing" containerID="cri-o://15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7" gracePeriod=30
Sep 17 23:24:25.908234 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:25.908208    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"]
Sep 17 23:24:25.908445 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:25.908422    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f" podUID="c3119a45-b6ed-4259-971f-7b2cfeb0fb60" containerName="payload-pre-processing" containerID="cri-o://08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b" gracePeriod=30
Sep 17 23:24:26.180103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.180076    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:24:26.183076 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.183057    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:24:26.314642 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.314612    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-plugins-config-volume\") pod \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") "
Sep 17 23:24:26.314833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.314655    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6c9k\" (UniqueName: \"kubernetes.io/projected/72374223-bb1b-4cbc-bc1a-5b2debcee787-kube-api-access-h6c9k\") pod \"72374223-bb1b-4cbc-bc1a-5b2debcee787\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") "
Sep 17 23:24:26.314833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.314707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfkr4\" (UniqueName: \"kubernetes.io/projected/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-kube-api-access-nfkr4\") pod \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\" (UID: \"c3119a45-b6ed-4259-971f-7b2cfeb0fb60\") "
Sep 17 23:24:26.314833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.314789    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72374223-bb1b-4cbc-bc1a-5b2debcee787-plugins-config-volume\") pod \"72374223-bb1b-4cbc-bc1a-5b2debcee787\" (UID: \"72374223-bb1b-4cbc-bc1a-5b2debcee787\") "
Sep 17 23:24:26.315009 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.314954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3119a45-b6ed-4259-971f-7b2cfeb0fb60" (UID: "c3119a45-b6ed-4259-971f-7b2cfeb0fb60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:26.315214 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.315185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72374223-bb1b-4cbc-bc1a-5b2debcee787-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72374223-bb1b-4cbc-bc1a-5b2debcee787" (UID: "72374223-bb1b-4cbc-bc1a-5b2debcee787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:26.316762 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.316741    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72374223-bb1b-4cbc-bc1a-5b2debcee787-kube-api-access-h6c9k" (OuterVolumeSpecName: "kube-api-access-h6c9k") pod "72374223-bb1b-4cbc-bc1a-5b2debcee787" (UID: "72374223-bb1b-4cbc-bc1a-5b2debcee787"). InnerVolumeSpecName "kube-api-access-h6c9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:26.316863 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.316847    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-kube-api-access-nfkr4" (OuterVolumeSpecName: "kube-api-access-nfkr4") pod "c3119a45-b6ed-4259-971f-7b2cfeb0fb60" (UID: "c3119a45-b6ed-4259-971f-7b2cfeb0fb60"). InnerVolumeSpecName "kube-api-access-nfkr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:26.415946 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.415893    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72374223-bb1b-4cbc-bc1a-5b2debcee787-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:24:26.415946 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.415913    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:24:26.415946 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.415923    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6c9k\" (UniqueName: \"kubernetes.io/projected/72374223-bb1b-4cbc-bc1a-5b2debcee787-kube-api-access-h6c9k\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:24:26.415946 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.415932    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkr4\" (UniqueName: \"kubernetes.io/projected/c3119a45-b6ed-4259-971f-7b2cfeb0fb60-kube-api-access-nfkr4\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:24:26.617094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.617059    2559 generic.go:356] "Generic (PLEG): container finished" podID="72374223-bb1b-4cbc-bc1a-5b2debcee787" containerID="15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7" exitCode=0
Sep 17 23:24:26.617254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.617112    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"
Sep 17 23:24:26.617254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.617144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv" event={"ID":"72374223-bb1b-4cbc-bc1a-5b2debcee787","Type":"ContainerDied","Data":"15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7"}
Sep 17 23:24:26.617254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.617205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv" event={"ID":"72374223-bb1b-4cbc-bc1a-5b2debcee787","Type":"ContainerDied","Data":"3a80613c0e80e57de609d3d921c7e5d8d871b7fe8c48709c4b6f3e40d62986f3"}
Sep 17 23:24:26.617254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.617236    2559 scope.go:117] "RemoveContainer" containerID="15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7"
Sep 17 23:24:26.618566 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.618544    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3119a45-b6ed-4259-971f-7b2cfeb0fb60" containerID="08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b" exitCode=0
Sep 17 23:24:26.618631 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.618581    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"
Sep 17 23:24:26.618631 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.618606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f" event={"ID":"c3119a45-b6ed-4259-971f-7b2cfeb0fb60","Type":"ContainerDied","Data":"08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b"}
Sep 17 23:24:26.618714 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.618646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f" event={"ID":"c3119a45-b6ed-4259-971f-7b2cfeb0fb60","Type":"ContainerDied","Data":"d4762d28e1c1188eedf3dfe5e0961f79eae0b423016e4c1eff48d6b06b902c64"}
Sep 17 23:24:26.630396 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.630241    2559 scope.go:117] "RemoveContainer" containerID="15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7"
Sep 17 23:24:26.630925 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:24:26.630900    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7\": container with ID starting with 15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7 not found: ID does not exist" containerID="15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7"
Sep 17 23:24:26.631003 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.630931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7"} err="failed to get container status \"15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7\": rpc error: code = NotFound desc = could not find container \"15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7\": container with ID starting with 15d625ac1eb6e10cade896abd443f4983af16450d377ed911e01a4022988d8f7 not found: ID does not exist"
Sep 17 23:24:26.631003 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.630949    2559 scope.go:117] "RemoveContainer" containerID="08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b"
Sep 17 23:24:26.639193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.639179    2559 scope.go:117] "RemoveContainer" containerID="08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b"
Sep 17 23:24:26.639439 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:24:26.639419    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b\": container with ID starting with 08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b not found: ID does not exist" containerID="08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b"
Sep 17 23:24:26.639520 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.639445    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b"} err="failed to get container status \"08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b\": rpc error: code = NotFound desc = could not find container \"08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b\": container with ID starting with 08672e8a5998432da90f8125d0e75153c281937b094e3f8b3f3b457a9dfbc44b not found: ID does not exist"
Sep 17 23:24:26.645736 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.645695    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"]
Sep 17 23:24:26.648764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.648744    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4f19d32d-68648674-prh6f"]
Sep 17 23:24:26.658821 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.658800    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"]
Sep 17 23:24:26.664268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:26.664249    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4f19d32d-5b87d9dd5c-p59jv"]
Sep 17 23:24:27.639841 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:27.639809    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72374223-bb1b-4cbc-bc1a-5b2debcee787" path="/var/lib/kubelet/pods/72374223-bb1b-4cbc-bc1a-5b2debcee787/volumes"
Sep 17 23:24:27.640214 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:27.640175    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3119a45-b6ed-4259-971f-7b2cfeb0fb60" path="/var/lib/kubelet/pods/c3119a45-b6ed-4259-971f-7b2cfeb0fb60/volumes"
Sep 17 23:24:30.737755 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:30.737710    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:30.738254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:30.737803    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:30.750393 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:30.750370    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:31.660238 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:31.660208    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:24:36.266200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:36.266158    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:24:47.233976 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.233943    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"]
Sep 17 23:24:47.234366 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.234354    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3119a45-b6ed-4259-971f-7b2cfeb0fb60" containerName="payload-pre-processing"
Sep 17 23:24:47.234411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.234367    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3119a45-b6ed-4259-971f-7b2cfeb0fb60" containerName="payload-pre-processing"
Sep 17 23:24:47.234411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.234380    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72374223-bb1b-4cbc-bc1a-5b2debcee787" containerName="payload-processing"
Sep 17 23:24:47.234411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.234386    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72374223-bb1b-4cbc-bc1a-5b2debcee787" containerName="payload-processing"
Sep 17 23:24:47.234522 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.234451    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3119a45-b6ed-4259-971f-7b2cfeb0fb60" containerName="payload-pre-processing"
Sep 17 23:24:47.234522 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.234462    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72374223-bb1b-4cbc-bc1a-5b2debcee787" containerName="payload-processing"
Sep 17 23:24:47.238847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.238829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.242119 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.242093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0c363f\"/\"test-model-0c363f-kserve-self-signed-certs\""
Sep 17 23:24:47.242266 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.242169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0c363f\"/\"openshift-service-ca.crt\""
Sep 17 23:24:47.242357 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.242337    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0c363f\"/\"kube-root-ca.crt\""
Sep 17 23:24:47.242527 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.242504    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0c363f\"/\"default-dockercfg-zpzb4\""
Sep 17 23:24:47.248180 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.248159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"]
Sep 17 23:24:47.283464 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.283445    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-dshm\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.283557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.283480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4nd\" (UniqueName: \"kubernetes.io/projected/70ef574f-b577-4145-b3c7-ec1f335d5ed2-kube-api-access-tp4nd\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.283557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.283518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-home\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.283557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.283539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tls-certs\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.283660 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.283564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tmp-dir\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.283660 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.283609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-model-cache\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384025 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.383997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tmp-dir\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-model-cache\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-dshm\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4nd\" (UniqueName: \"kubernetes.io/projected/70ef574f-b577-4145-b3c7-ec1f335d5ed2-kube-api-access-tp4nd\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-home\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384358 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tls-certs\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384443 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tmp-dir\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384511 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-model-cache\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.384633 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.384612    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-home\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.386314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.386292    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-dshm\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.386603 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.386587    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tls-certs\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.392213 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.392186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4nd\" (UniqueName: \"kubernetes.io/projected/70ef574f-b577-4145-b3c7-ec1f335d5ed2-kube-api-access-tp4nd\") pod \"test-model-0c363f-kserve-7d79f55cf5-nr756\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") " pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.549393 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.549320    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:47.822028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:47.821958    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"]
Sep 17 23:24:47.825427 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:24:47.825400    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ef574f_b577_4145_b3c7_ec1f335d5ed2.slice/crio-365063cb2af49102da002fca4efd732e36f494614608c4ff9299dd565fba5cf5 WatchSource:0}: Error finding container 365063cb2af49102da002fca4efd732e36f494614608c4ff9299dd565fba5cf5: Status 404 returned error can't find the container with id 365063cb2af49102da002fca4efd732e36f494614608c4ff9299dd565fba5cf5
Sep 17 23:24:48.742058 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:48.742025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756" event={"ID":"70ef574f-b577-4145-b3c7-ec1f335d5ed2","Type":"ContainerStarted","Data":"be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48"}
Sep 17 23:24:48.742405 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:48.742066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756" event={"ID":"70ef574f-b577-4145-b3c7-ec1f335d5ed2","Type":"ContainerStarted","Data":"365063cb2af49102da002fca4efd732e36f494614608c4ff9299dd565fba5cf5"}
Sep 17 23:24:48.757980 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:48.757936    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756" podStartSLOduration=1.757921818 podStartE2EDuration="1.757921818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:48.757001552 +0000 UTC m=+1131.230944534" watchObservedRunningTime="2026-09-17 23:24:48.757921818 +0000 UTC m=+1131.231864800"
Sep 17 23:24:56.438331 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:56.438288    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:24:57.549554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:57.549518    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:57.549992 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:57.549588    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:57.562748 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:57.562708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:57.799627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:57.799563    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:24:58.807552 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.807520    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"]
Sep 17 23:24:58.811456 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.811438    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:24:58.813977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.813954    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:24:58.814083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.814008    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\""
Sep 17 23:24:58.820175 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.820038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"]
Sep 17 23:24:58.884679 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.884653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f57641-5157-440e-b189-d10f9325c77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2\" (UID: \"72f57641-5157-440e-b189-d10f9325c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:24:58.985524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.985495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f57641-5157-440e-b189-d10f9325c77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2\" (UID: \"72f57641-5157-440e-b189-d10f9325c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:24:58.986072 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:58.986055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f57641-5157-440e-b189-d10f9325c77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2\" (UID: \"72f57641-5157-440e-b189-d10f9325c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:24:59.131361 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:59.131303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:24:59.405383 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:59.405209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"]
Sep 17 23:24:59.407705 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:24:59.407673    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f57641_5157_440e_b189_d10f9325c77e.slice/crio-bfbb5f8b885eb76eb3ddbe61d55a67cb372e6785fd0f29da88bfb528a4e5c578 WatchSource:0}: Error finding container bfbb5f8b885eb76eb3ddbe61d55a67cb372e6785fd0f29da88bfb528a4e5c578: Status 404 returned error can't find the container with id bfbb5f8b885eb76eb3ddbe61d55a67cb372e6785fd0f29da88bfb528a4e5c578
Sep 17 23:24:59.799436 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:24:59.799404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2" event={"ID":"72f57641-5157-440e-b189-d10f9325c77e","Type":"ContainerStarted","Data":"bfbb5f8b885eb76eb3ddbe61d55a67cb372e6785fd0f29da88bfb528a4e5c578"}
Sep 17 23:25:03.619935 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:03.619890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"]
Sep 17 23:25:04.499763 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.499713    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"]
Sep 17 23:25:04.499976 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.499929    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4" podUID="ad4df59f-f3b1-405e-954e-03b460a06bcc" containerName="payload-processing" containerID="cri-o://5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f" gracePeriod=30
Sep 17 23:25:04.761608 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.761540    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:25:04.826329 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.826294    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad4df59f-f3b1-405e-954e-03b460a06bcc" containerID="5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f" exitCode=0
Sep 17 23:25:04.826472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.826341    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"
Sep 17 23:25:04.826472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.826383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4" event={"ID":"ad4df59f-f3b1-405e-954e-03b460a06bcc","Type":"ContainerDied","Data":"5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f"}
Sep 17 23:25:04.826472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.826446    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4" event={"ID":"ad4df59f-f3b1-405e-954e-03b460a06bcc","Type":"ContainerDied","Data":"79eb822992e19a9bcb5f0932c2de96cca96b98e310504a6fb1dd0234fdb44ca1"}
Sep 17 23:25:04.826594 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.826480    2559 scope.go:117] "RemoveContainer" containerID="5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f"
Sep 17 23:25:04.831840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.831816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4df59f-f3b1-405e-954e-03b460a06bcc-plugins-config-volume\") pod \"ad4df59f-f3b1-405e-954e-03b460a06bcc\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") "
Sep 17 23:25:04.831958 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.831913    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgn4\" (UniqueName: \"kubernetes.io/projected/ad4df59f-f3b1-405e-954e-03b460a06bcc-kube-api-access-8wgn4\") pod \"ad4df59f-f3b1-405e-954e-03b460a06bcc\" (UID: \"ad4df59f-f3b1-405e-954e-03b460a06bcc\") "
Sep 17 23:25:04.832187 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.832165    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4df59f-f3b1-405e-954e-03b460a06bcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad4df59f-f3b1-405e-954e-03b460a06bcc" (UID: "ad4df59f-f3b1-405e-954e-03b460a06bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:04.832289 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.832275    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4df59f-f3b1-405e-954e-03b460a06bcc-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:04.833921 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.833894    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4df59f-f3b1-405e-954e-03b460a06bcc-kube-api-access-8wgn4" (OuterVolumeSpecName: "kube-api-access-8wgn4") pod "ad4df59f-f3b1-405e-954e-03b460a06bcc" (UID: "ad4df59f-f3b1-405e-954e-03b460a06bcc"). InnerVolumeSpecName "kube-api-access-8wgn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:04.835622 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.835599    2559 scope.go:117] "RemoveContainer" containerID="5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f"
Sep 17 23:25:04.835975 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:25:04.835952    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f\": container with ID starting with 5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f not found: ID does not exist" containerID="5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f"
Sep 17 23:25:04.836038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.835987    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f"} err="failed to get container status \"5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f\": rpc error: code = NotFound desc = could not find container \"5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f\": container with ID starting with 5ba3a458339f655131628bb3b0b28a5bebcb28a7b844ae8ff6daf13a8466916f not found: ID does not exist"
Sep 17 23:25:04.932999 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:04.932980    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgn4\" (UniqueName: \"kubernetes.io/projected/ad4df59f-f3b1-405e-954e-03b460a06bcc-kube-api-access-8wgn4\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:05.150906 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:05.150880    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"]
Sep 17 23:25:05.155093 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:05.155072    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63b144-dc7fdfbc7-w9xw4"]
Sep 17 23:25:05.640091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:05.640062    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad4df59f-f3b1-405e-954e-03b460a06bcc" path="/var/lib/kubelet/pods/ad4df59f-f3b1-405e-954e-03b460a06bcc/volumes"
Sep 17 23:25:09.169169 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.169119    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"]
Sep 17 23:25:09.169700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.169657    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57" podUID="237aa454-21cb-4841-9f6a-45bd31700e9c" containerName="payload-processing" containerID="cri-o://79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb" gracePeriod=30
Sep 17 23:25:09.177656 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.177624    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"]
Sep 17 23:25:09.177901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.177867    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w" podUID="fe2856ab-8c97-423a-8487-ec62b3989483" containerName="payload-pre-processing" containerID="cri-o://812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5" gracePeriod=30
Sep 17 23:25:09.449411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.449391    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:25:09.452021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.452002    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:25:09.571690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.571662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxrfm\" (UniqueName: \"kubernetes.io/projected/fe2856ab-8c97-423a-8487-ec62b3989483-kube-api-access-gxrfm\") pod \"fe2856ab-8c97-423a-8487-ec62b3989483\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") "
Sep 17 23:25:09.571827 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.571779    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237aa454-21cb-4841-9f6a-45bd31700e9c-plugins-config-volume\") pod \"237aa454-21cb-4841-9f6a-45bd31700e9c\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") "
Sep 17 23:25:09.571873 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.571824    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6x8z\" (UniqueName: \"kubernetes.io/projected/237aa454-21cb-4841-9f6a-45bd31700e9c-kube-api-access-k6x8z\") pod \"237aa454-21cb-4841-9f6a-45bd31700e9c\" (UID: \"237aa454-21cb-4841-9f6a-45bd31700e9c\") "
Sep 17 23:25:09.571945 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.571916    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2856ab-8c97-423a-8487-ec62b3989483-plugins-config-volume\") pod \"fe2856ab-8c97-423a-8487-ec62b3989483\" (UID: \"fe2856ab-8c97-423a-8487-ec62b3989483\") "
Sep 17 23:25:09.572183 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.572160    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237aa454-21cb-4841-9f6a-45bd31700e9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "237aa454-21cb-4841-9f6a-45bd31700e9c" (UID: "237aa454-21cb-4841-9f6a-45bd31700e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:09.572250 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.572201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2856ab-8c97-423a-8487-ec62b3989483-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe2856ab-8c97-423a-8487-ec62b3989483" (UID: "fe2856ab-8c97-423a-8487-ec62b3989483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:09.573805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.573784    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2856ab-8c97-423a-8487-ec62b3989483-kube-api-access-gxrfm" (OuterVolumeSpecName: "kube-api-access-gxrfm") pod "fe2856ab-8c97-423a-8487-ec62b3989483" (UID: "fe2856ab-8c97-423a-8487-ec62b3989483"). InnerVolumeSpecName "kube-api-access-gxrfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:09.573994 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.573973    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237aa454-21cb-4841-9f6a-45bd31700e9c-kube-api-access-k6x8z" (OuterVolumeSpecName: "kube-api-access-k6x8z") pod "237aa454-21cb-4841-9f6a-45bd31700e9c" (UID: "237aa454-21cb-4841-9f6a-45bd31700e9c"). InnerVolumeSpecName "kube-api-access-k6x8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:09.673051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.672981    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2856ab-8c97-423a-8487-ec62b3989483-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:09.673051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.673006    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrfm\" (UniqueName: \"kubernetes.io/projected/fe2856ab-8c97-423a-8487-ec62b3989483-kube-api-access-gxrfm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:09.673051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.673020    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237aa454-21cb-4841-9f6a-45bd31700e9c-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:09.673051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.673033    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6x8z\" (UniqueName: \"kubernetes.io/projected/237aa454-21cb-4841-9f6a-45bd31700e9c-kube-api-access-k6x8z\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:09.855703 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.855669    2559 generic.go:356] "Generic (PLEG): container finished" podID="fe2856ab-8c97-423a-8487-ec62b3989483" containerID="812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5" exitCode=0
Sep 17 23:25:09.855892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.855745    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"
Sep 17 23:25:09.855892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.855750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w" event={"ID":"fe2856ab-8c97-423a-8487-ec62b3989483","Type":"ContainerDied","Data":"812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5"}
Sep 17 23:25:09.855892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.855815    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w" event={"ID":"fe2856ab-8c97-423a-8487-ec62b3989483","Type":"ContainerDied","Data":"e764b6cbbe758e082a5dcc1e45d8c01868447406e4efa1c42d6db7d5f0325c3c"}
Sep 17 23:25:09.855892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.855852    2559 scope.go:117] "RemoveContainer" containerID="812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5"
Sep 17 23:25:09.857158 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.857132    2559 generic.go:356] "Generic (PLEG): container finished" podID="237aa454-21cb-4841-9f6a-45bd31700e9c" containerID="79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb" exitCode=0
Sep 17 23:25:09.857296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.857183    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"
Sep 17 23:25:09.857296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.857217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57" event={"ID":"237aa454-21cb-4841-9f6a-45bd31700e9c","Type":"ContainerDied","Data":"79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb"}
Sep 17 23:25:09.857296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.857284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57" event={"ID":"237aa454-21cb-4841-9f6a-45bd31700e9c","Type":"ContainerDied","Data":"ebec4c7e38be0bea3b941514b73108de88382b399d12b3734372895325500f71"}
Sep 17 23:25:09.864656 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.864635    2559 scope.go:117] "RemoveContainer" containerID="812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5"
Sep 17 23:25:09.864935 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:25:09.864913    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5\": container with ID starting with 812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5 not found: ID does not exist" containerID="812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5"
Sep 17 23:25:09.865012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.864943    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5"} err="failed to get container status \"812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5\": rpc error: code = NotFound desc = could not find container \"812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5\": container with ID starting with 812cf53fdb77ec1482e86d5156e66fb17ce88117243d931a902a3328b52734a5 not found: ID does not exist"
Sep 17 23:25:09.865012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.864958    2559 scope.go:117] "RemoveContainer" containerID="79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb"
Sep 17 23:25:09.872993 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.872972    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"]
Sep 17 23:25:09.873587 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.873570    2559 scope.go:117] "RemoveContainer" containerID="79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb"
Sep 17 23:25:09.873857 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:25:09.873834    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb\": container with ID starting with 79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb not found: ID does not exist" containerID="79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb"
Sep 17 23:25:09.873938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.873860    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb"} err="failed to get container status \"79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb\": rpc error: code = NotFound desc = could not find container \"79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb\": container with ID starting with 79c1d15036ad79e66d1832737047f4c3befce5d0682c84abf357db837f46d9fb not found: ID does not exist"
Sep 17 23:25:09.877521 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.877497    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-983490-9959f9ccc-bjg57"]
Sep 17 23:25:09.888529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.888509    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"]
Sep 17 23:25:09.891911 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.891892    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-983490-7cd5779d86-dbm8w"]
Sep 17 23:25:09.996124 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:09.996099    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"]
Sep 17 23:25:11.645867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:11.641317    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="237aa454-21cb-4841-9f6a-45bd31700e9c" path="/var/lib/kubelet/pods/237aa454-21cb-4841-9f6a-45bd31700e9c/volumes"
Sep 17 23:25:11.645867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:11.641904    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2856ab-8c97-423a-8487-ec62b3989483" path="/var/lib/kubelet/pods/fe2856ab-8c97-423a-8487-ec62b3989483/volumes"
Sep 17 23:25:13.882800 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:13.882739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2" event={"ID":"72f57641-5157-440e-b189-d10f9325c77e","Type":"ContainerStarted","Data":"1972259ab01841174bd7c93be7bb83ccc11cda256afd269a524c85495b10ee7a"}
Sep 17 23:25:13.899256 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:13.899214    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2" podStartSLOduration=2.099589335 podStartE2EDuration="15.899199903s" totalImagesPullingTime="13.799610568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:13.897421794 +0000 UTC m=+1156.371364779" watchObservedRunningTime="2026-09-17 23:25:13.899199903 +0000 UTC m=+1156.373142921"
Sep 17 23:25:14.891051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:14.891018    2559 generic.go:356] "Generic (PLEG): container finished" podID="72f57641-5157-440e-b189-d10f9325c77e" containerID="1972259ab01841174bd7c93be7bb83ccc11cda256afd269a524c85495b10ee7a" exitCode=0
Sep 17 23:25:14.891395 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:14.891086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2" event={"ID":"72f57641-5157-440e-b189-d10f9325c77e","Type":"ContainerDied","Data":"1972259ab01841174bd7c93be7bb83ccc11cda256afd269a524c85495b10ee7a"}
Sep 17 23:25:16.029881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.029850    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:25:16.135177 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.135147    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f57641-5157-440e-b189-d10f9325c77e-maas-api-service-ca\") pod \"72f57641-5157-440e-b189-d10f9325c77e\" (UID: \"72f57641-5157-440e-b189-d10f9325c77e\") "
Sep 17 23:25:16.135489 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.135466    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f57641-5157-440e-b189-d10f9325c77e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72f57641-5157-440e-b189-d10f9325c77e" (UID: "72f57641-5157-440e-b189-d10f9325c77e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:16.236350 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.236327    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f57641-5157-440e-b189-d10f9325c77e-maas-api-service-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:16.903067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.903032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2" event={"ID":"72f57641-5157-440e-b189-d10f9325c77e","Type":"ContainerDied","Data":"bfbb5f8b885eb76eb3ddbe61d55a67cb372e6785fd0f29da88bfb528a4e5c578"}
Sep 17 23:25:16.903067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.903053    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"
Sep 17 23:25:16.903262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:16.903083    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfbb5f8b885eb76eb3ddbe61d55a67cb372e6785fd0f29da88bfb528a4e5c578"
Sep 17 23:25:18.640659 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:18.640610    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz" podUID="31cc1d20-7e27-4b7e-9184-e47ad4539b57" containerName="main" containerID="cri-o://9722f78b0623b3450aba3feb1d1dd6fde05fffff909169c220f5628ac8b37762" gracePeriod=45
Sep 17 23:25:18.915198 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:18.915165    2559 generic.go:356] "Generic (PLEG): container finished" podID="31cc1d20-7e27-4b7e-9184-e47ad4539b57" containerID="9722f78b0623b3450aba3feb1d1dd6fde05fffff909169c220f5628ac8b37762" exitCode=0
Sep 17 23:25:18.915324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:18.915236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz" event={"ID":"31cc1d20-7e27-4b7e-9184-e47ad4539b57","Type":"ContainerDied","Data":"9722f78b0623b3450aba3feb1d1dd6fde05fffff909169c220f5628ac8b37762"}
Sep 17 23:25:18.915324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:18.915286    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz" event={"ID":"31cc1d20-7e27-4b7e-9184-e47ad4539b57","Type":"ContainerDied","Data":"58a844562cc30d10e2d93f4e5aae5bdaf8dcb00e1707c11beddcaa547a386918"}
Sep 17 23:25:18.915324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:18.915301    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a844562cc30d10e2d93f4e5aae5bdaf8dcb00e1707c11beddcaa547a386918"
Sep 17 23:25:18.933372 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:18.933344    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:25:19.063321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063289    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-model-cache\") pod \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") "
Sep 17 23:25:19.063466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063339    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-dshm\") pod \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") "
Sep 17 23:25:19.063466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-home\") pod \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") "
Sep 17 23:25:19.063466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063424    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tls-certs\") pod \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") "
Sep 17 23:25:19.063466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2z48\" (UniqueName: \"kubernetes.io/projected/31cc1d20-7e27-4b7e-9184-e47ad4539b57-kube-api-access-z2z48\") pod \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") "
Sep 17 23:25:19.063699 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063513    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tmp-dir\") pod \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\" (UID: \"31cc1d20-7e27-4b7e-9184-e47ad4539b57\") "
Sep 17 23:25:19.063699 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-model-cache" (OuterVolumeSpecName: "model-cache") pod "31cc1d20-7e27-4b7e-9184-e47ad4539b57" (UID: "31cc1d20-7e27-4b7e-9184-e47ad4539b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:19.063699 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063682    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-home" (OuterVolumeSpecName: "home") pod "31cc1d20-7e27-4b7e-9184-e47ad4539b57" (UID: "31cc1d20-7e27-4b7e-9184-e47ad4539b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:19.063873 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063818    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31cc1d20-7e27-4b7e-9184-e47ad4539b57" (UID: "31cc1d20-7e27-4b7e-9184-e47ad4539b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:19.063925 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063872    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:19.063925 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.063891    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:19.065689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.065662    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cc1d20-7e27-4b7e-9184-e47ad4539b57-kube-api-access-z2z48" (OuterVolumeSpecName: "kube-api-access-z2z48") pod "31cc1d20-7e27-4b7e-9184-e47ad4539b57" (UID: "31cc1d20-7e27-4b7e-9184-e47ad4539b57"). InnerVolumeSpecName "kube-api-access-z2z48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:19.065965 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.065942    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-dshm" (OuterVolumeSpecName: "dshm") pod "31cc1d20-7e27-4b7e-9184-e47ad4539b57" (UID: "31cc1d20-7e27-4b7e-9184-e47ad4539b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:19.066045 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.065943    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31cc1d20-7e27-4b7e-9184-e47ad4539b57" (UID: "31cc1d20-7e27-4b7e-9184-e47ad4539b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:19.165160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.165102    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:19.165160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.165124    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc1d20-7e27-4b7e-9184-e47ad4539b57-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:19.165160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.165133    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc1d20-7e27-4b7e-9184-e47ad4539b57-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:19.165160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.165142    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2z48\" (UniqueName: \"kubernetes.io/projected/31cc1d20-7e27-4b7e-9184-e47ad4539b57-kube-api-access-z2z48\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:19.642731 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.642687    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"]
Sep 17 23:25:19.643115 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.642894    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b" podUID="02a1c5c8-817c-4d8c-bd12-b8b69fea2610" containerName="payload-processing" containerID="cri-o://68d58a2e74c0eb50aba638607b166e4084a4f1142913b15614cd153b8d6082c5" gracePeriod=30
Sep 17 23:25:19.655475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.655449    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"]
Sep 17 23:25:19.655664 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.655644    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp" podUID="39d7df89-adda-4ccd-b183-1e91d10023a0" containerName="payload-pre-processing" containerID="cri-o://d1441a4330f6811d9fc2dae1b52f5dd1f2abc87546ad6f4709929c10226bdcb8" gracePeriod=30
Sep 17 23:25:19.835172 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.835141    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"]
Sep 17 23:25:19.838245 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.838222    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-861522-wb7b2"]
Sep 17 23:25:19.922730 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.922689    2559 generic.go:356] "Generic (PLEG): container finished" podID="39d7df89-adda-4ccd-b183-1e91d10023a0" containerID="d1441a4330f6811d9fc2dae1b52f5dd1f2abc87546ad6f4709929c10226bdcb8" exitCode=0
Sep 17 23:25:19.922842 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.922752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp" event={"ID":"39d7df89-adda-4ccd-b183-1e91d10023a0","Type":"ContainerDied","Data":"d1441a4330f6811d9fc2dae1b52f5dd1f2abc87546ad6f4709929c10226bdcb8"}
Sep 17 23:25:19.925625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.924877    2559 generic.go:356] "Generic (PLEG): container finished" podID="02a1c5c8-817c-4d8c-bd12-b8b69fea2610" containerID="68d58a2e74c0eb50aba638607b166e4084a4f1142913b15614cd153b8d6082c5" exitCode=0
Sep 17 23:25:19.925625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.925022    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"
Sep 17 23:25:19.925625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.925280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b" event={"ID":"02a1c5c8-817c-4d8c-bd12-b8b69fea2610","Type":"ContainerDied","Data":"68d58a2e74c0eb50aba638607b166e4084a4f1142913b15614cd153b8d6082c5"}
Sep 17 23:25:19.925625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.925340    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b" event={"ID":"02a1c5c8-817c-4d8c-bd12-b8b69fea2610","Type":"ContainerDied","Data":"a2f2cd3b2fbad025686aced8cb8e741bcdaf29c878496e71507f9320ac0919ae"}
Sep 17 23:25:19.925625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.925363    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2f2cd3b2fbad025686aced8cb8e741bcdaf29c878496e71507f9320ac0919ae"
Sep 17 23:25:19.941366 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.941344    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:25:19.944473 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.944455    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:25:19.956968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.956934    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"]
Sep 17 23:25:19.958476 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:19.958454    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-861522/simulator-w0-861522-kserve-c4ffb5f4f-l62vz"]
Sep 17 23:25:20.074132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.074094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjzbv\" (UniqueName: \"kubernetes.io/projected/39d7df89-adda-4ccd-b183-1e91d10023a0-kube-api-access-kjzbv\") pod \"39d7df89-adda-4ccd-b183-1e91d10023a0\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") "
Sep 17 23:25:20.074270 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.074153    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vxpk\" (UniqueName: \"kubernetes.io/projected/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-kube-api-access-2vxpk\") pod \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") "
Sep 17 23:25:20.074484 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.074382    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-plugins-config-volume\") pod \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\" (UID: \"02a1c5c8-817c-4d8c-bd12-b8b69fea2610\") "
Sep 17 23:25:20.074484 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.074456    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7df89-adda-4ccd-b183-1e91d10023a0-plugins-config-volume\") pod \"39d7df89-adda-4ccd-b183-1e91d10023a0\" (UID: \"39d7df89-adda-4ccd-b183-1e91d10023a0\") "
Sep 17 23:25:20.074841 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.074818    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02a1c5c8-817c-4d8c-bd12-b8b69fea2610" (UID: "02a1c5c8-817c-4d8c-bd12-b8b69fea2610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:20.075037 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.075016    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d7df89-adda-4ccd-b183-1e91d10023a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39d7df89-adda-4ccd-b183-1e91d10023a0" (UID: "39d7df89-adda-4ccd-b183-1e91d10023a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:20.076426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.076402    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d7df89-adda-4ccd-b183-1e91d10023a0-kube-api-access-kjzbv" (OuterVolumeSpecName: "kube-api-access-kjzbv") pod "39d7df89-adda-4ccd-b183-1e91d10023a0" (UID: "39d7df89-adda-4ccd-b183-1e91d10023a0"). InnerVolumeSpecName "kube-api-access-kjzbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:20.076524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.076506    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-kube-api-access-2vxpk" (OuterVolumeSpecName: "kube-api-access-2vxpk") pod "02a1c5c8-817c-4d8c-bd12-b8b69fea2610" (UID: "02a1c5c8-817c-4d8c-bd12-b8b69fea2610"). InnerVolumeSpecName "kube-api-access-2vxpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:20.175401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.175348    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzbv\" (UniqueName: \"kubernetes.io/projected/39d7df89-adda-4ccd-b183-1e91d10023a0-kube-api-access-kjzbv\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:20.175401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.175369    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxpk\" (UniqueName: \"kubernetes.io/projected/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-kube-api-access-2vxpk\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:20.175401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.175380    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a1c5c8-817c-4d8c-bd12-b8b69fea2610-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:20.175401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.175388    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d7df89-adda-4ccd-b183-1e91d10023a0-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:20.932035 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.931972    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"
Sep 17 23:25:20.932035 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.931991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp" event={"ID":"39d7df89-adda-4ccd-b183-1e91d10023a0","Type":"ContainerDied","Data":"42f05485657ad457a7b7404758092b4c0fb057d588a8f83565d666402bef6060"}
Sep 17 23:25:20.932509 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.932045    2559 scope.go:117] "RemoveContainer" containerID="d1441a4330f6811d9fc2dae1b52f5dd1f2abc87546ad6f4709929c10226bdcb8"
Sep 17 23:25:20.932509 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.931974    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"
Sep 17 23:25:20.957354 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.957333    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"]
Sep 17 23:25:20.959181 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.959160    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-861522-7f595b7ff-fnl7b"]
Sep 17 23:25:20.970915 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.970894    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"]
Sep 17 23:25:20.975497 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:20.975471    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-861522-6c767f67c4-knlqp"]
Sep 17 23:25:21.140591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:21.140565    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:25:21.640115 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:21.640085    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a1c5c8-817c-4d8c-bd12-b8b69fea2610" path="/var/lib/kubelet/pods/02a1c5c8-817c-4d8c-bd12-b8b69fea2610/volumes"
Sep 17 23:25:21.640523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:21.640507    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cc1d20-7e27-4b7e-9184-e47ad4539b57" path="/var/lib/kubelet/pods/31cc1d20-7e27-4b7e-9184-e47ad4539b57/volumes"
Sep 17 23:25:21.640911 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:21.640895    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d7df89-adda-4ccd-b183-1e91d10023a0" path="/var/lib/kubelet/pods/39d7df89-adda-4ccd-b183-1e91d10023a0/volumes"
Sep 17 23:25:21.641246 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:21.641232    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f57641-5157-440e-b189-d10f9325c77e" path="/var/lib/kubelet/pods/72f57641-5157-440e-b189-d10f9325c77e/volumes"
Sep 17 23:25:25.015968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.015921    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2" podUID="39b5bb69-edce-497a-81d1-9e1ab8821a56" containerName="main" containerID="cri-o://30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad" gracePeriod=45
Sep 17 23:25:25.259491 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.259464    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:25:25.421036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.420963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-model-cache\") pod \"39b5bb69-edce-497a-81d1-9e1ab8821a56\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") "
Sep 17 23:25:25.421036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421006    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c65vk\" (UniqueName: \"kubernetes.io/projected/39b5bb69-edce-497a-81d1-9e1ab8821a56-kube-api-access-c65vk\") pod \"39b5bb69-edce-497a-81d1-9e1ab8821a56\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") "
Sep 17 23:25:25.421036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421028    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-dshm\") pod \"39b5bb69-edce-497a-81d1-9e1ab8821a56\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") "
Sep 17 23:25:25.421277 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421059    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-home\") pod \"39b5bb69-edce-497a-81d1-9e1ab8821a56\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") "
Sep 17 23:25:25.421277 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421097    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-tmp-dir\") pod \"39b5bb69-edce-497a-81d1-9e1ab8821a56\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") "
Sep 17 23:25:25.421277 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421135    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b5bb69-edce-497a-81d1-9e1ab8821a56-tls-certs\") pod \"39b5bb69-edce-497a-81d1-9e1ab8821a56\" (UID: \"39b5bb69-edce-497a-81d1-9e1ab8821a56\") "
Sep 17 23:25:25.421277 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-model-cache" (OuterVolumeSpecName: "model-cache") pod "39b5bb69-edce-497a-81d1-9e1ab8821a56" (UID: "39b5bb69-edce-497a-81d1-9e1ab8821a56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:25.421468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421397    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-home" (OuterVolumeSpecName: "home") pod "39b5bb69-edce-497a-81d1-9e1ab8821a56" (UID: "39b5bb69-edce-497a-81d1-9e1ab8821a56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:25.421468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421416    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39b5bb69-edce-497a-81d1-9e1ab8821a56" (UID: "39b5bb69-edce-497a-81d1-9e1ab8821a56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:25.421546 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.421474    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.423264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.423236    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-dshm" (OuterVolumeSpecName: "dshm") pod "39b5bb69-edce-497a-81d1-9e1ab8821a56" (UID: "39b5bb69-edce-497a-81d1-9e1ab8821a56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:25.423376 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.423358    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b5bb69-edce-497a-81d1-9e1ab8821a56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39b5bb69-edce-497a-81d1-9e1ab8821a56" (UID: "39b5bb69-edce-497a-81d1-9e1ab8821a56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:25.423434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.423359    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b5bb69-edce-497a-81d1-9e1ab8821a56-kube-api-access-c65vk" (OuterVolumeSpecName: "kube-api-access-c65vk") pod "39b5bb69-edce-497a-81d1-9e1ab8821a56" (UID: "39b5bb69-edce-497a-81d1-9e1ab8821a56"). InnerVolumeSpecName "kube-api-access-c65vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:25.522504 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.522479    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.522504 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.522502    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.522645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.522513    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b5bb69-edce-497a-81d1-9e1ab8821a56-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.522645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.522522    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65vk\" (UniqueName: \"kubernetes.io/projected/39b5bb69-edce-497a-81d1-9e1ab8821a56-kube-api-access-c65vk\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.522645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.522530    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b5bb69-edce-497a-81d1-9e1ab8821a56-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.960428 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.960403    2559 generic.go:356] "Generic (PLEG): container finished" podID="39b5bb69-edce-497a-81d1-9e1ab8821a56" containerID="30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad" exitCode=0
Sep 17 23:25:25.960560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.960449    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"
Sep 17 23:25:25.960560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.960489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2" event={"ID":"39b5bb69-edce-497a-81d1-9e1ab8821a56","Type":"ContainerDied","Data":"30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad"}
Sep 17 23:25:25.960560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.960535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2" event={"ID":"39b5bb69-edce-497a-81d1-9e1ab8821a56","Type":"ContainerDied","Data":"b4e7a4eb5f7b8c46455ac5f73fc9fbdd3dfab6f3c02eea206671874d1e5fb628"}
Sep 17 23:25:25.960560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.960553    2559 scope.go:117] "RemoveContainer" containerID="30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad"
Sep 17 23:25:25.970026 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.970005    2559 scope.go:117] "RemoveContainer" containerID="30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad"
Sep 17 23:25:25.970339 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:25:25.970320    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad\": container with ID starting with 30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad not found: ID does not exist" containerID="30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad"
Sep 17 23:25:25.970441 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.970349    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad"} err="failed to get container status \"30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad\": rpc error: code = NotFound desc = could not find container \"30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad\": container with ID starting with 30ba1ad42827ce650ab9623e66c0027f85179b87adba948a61fcefa3a7996fad not found: ID does not exist"
Sep 17 23:25:25.981929 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.981904    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"]
Sep 17 23:25:25.986240 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:25.986217    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-63b144/autores-model-63b144-kserve-55c9c966f5-nsvz2"]
Sep 17 23:25:27.638988 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:27.638957    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b5bb69-edce-497a-81d1-9e1ab8821a56" path="/var/lib/kubelet/pods/39b5bb69-edce-497a-81d1-9e1ab8821a56/volumes"
Sep 17 23:25:37.140912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:37.140873    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:25:46.612831 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.612797    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"]
Sep 17 23:25:46.613190 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613178    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d7df89-adda-4ccd-b183-1e91d10023a0" containerName="payload-pre-processing"
Sep 17 23:25:46.613190 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613188    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d7df89-adda-4ccd-b183-1e91d10023a0" containerName="payload-pre-processing"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613197    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="237aa454-21cb-4841-9f6a-45bd31700e9c" containerName="payload-processing"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613203    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="237aa454-21cb-4841-9f6a-45bd31700e9c" containerName="payload-processing"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613209    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f57641-5157-440e-b189-d10f9325c77e" containerName="revoke-keys"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613215    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f57641-5157-440e-b189-d10f9325c77e" containerName="revoke-keys"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613227    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b5bb69-edce-497a-81d1-9e1ab8821a56" containerName="main"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613232    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b5bb69-edce-497a-81d1-9e1ab8821a56" containerName="main"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613239    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2856ab-8c97-423a-8487-ec62b3989483" containerName="payload-pre-processing"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613244    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2856ab-8c97-423a-8487-ec62b3989483" containerName="payload-pre-processing"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613253    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cc1d20-7e27-4b7e-9184-e47ad4539b57" containerName="main"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613258    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cc1d20-7e27-4b7e-9184-e47ad4539b57" containerName="main"
Sep 17 23:25:46.613264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613268    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4df59f-f3b1-405e-954e-03b460a06bcc" containerName="payload-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613273    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4df59f-f3b1-405e-954e-03b460a06bcc" containerName="payload-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613282    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a1c5c8-817c-4d8c-bd12-b8b69fea2610" containerName="payload-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613287    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a1c5c8-817c-4d8c-bd12-b8b69fea2610" containerName="payload-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613336    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2856ab-8c97-423a-8487-ec62b3989483" containerName="payload-pre-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613344    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b5bb69-edce-497a-81d1-9e1ab8821a56" containerName="main"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613352    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d7df89-adda-4ccd-b183-1e91d10023a0" containerName="payload-pre-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613357    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cc1d20-7e27-4b7e-9184-e47ad4539b57" containerName="main"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613365    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f57641-5157-440e-b189-d10f9325c77e" containerName="revoke-keys"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613372    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a1c5c8-817c-4d8c-bd12-b8b69fea2610" containerName="payload-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613377    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="237aa454-21cb-4841-9f6a-45bd31700e9c" containerName="payload-processing"
Sep 17 23:25:46.613592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.613382    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad4df59f-f3b1-405e-954e-03b460a06bcc" containerName="payload-processing"
Sep 17 23:25:46.621407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.621390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.624274 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.624196    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-307a7b\"/\"default-dockercfg-zxgcf\""
Sep 17 23:25:46.624411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.624389    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-307a7b\"/\"kube-root-ca.crt\""
Sep 17 23:25:46.624479 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.624436    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-307a7b\"/\"openshift-service-ca.crt\""
Sep 17 23:25:46.624534 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.624520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-307a7b\"/\"test-model-307a7b-kserve-self-signed-certs\""
Sep 17 23:25:46.625820 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.625790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"]
Sep 17 23:25:46.690519 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.690495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e473c71a-5795-4b39-9737-0d92441c33ee-tls-certs\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.690613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.690528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9rp\" (UniqueName: \"kubernetes.io/projected/e473c71a-5795-4b39-9737-0d92441c33ee-kube-api-access-tr9rp\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.690613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.690553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-home\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.690613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.690596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-model-cache\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.690748 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.690618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-tmp-dir\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.690748 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.690651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-dshm\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.791931 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.791894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-home\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792106 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.791935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-model-cache\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792106 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.791973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-tmp-dir\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792106 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.792012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-dshm\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792278 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.792111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e473c71a-5795-4b39-9737-0d92441c33ee-tls-certs\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792278 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.792146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9rp\" (UniqueName: \"kubernetes.io/projected/e473c71a-5795-4b39-9737-0d92441c33ee-kube-api-access-tr9rp\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.792322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-home\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.792364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-model-cache\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.792484 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.792392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-tmp-dir\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.794298 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.794266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-dshm\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.794675 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.794655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e473c71a-5795-4b39-9737-0d92441c33ee-tls-certs\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.804479 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.804456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9rp\" (UniqueName: \"kubernetes.io/projected/e473c71a-5795-4b39-9737-0d92441c33ee-kube-api-access-tr9rp\") pod \"test-model-307a7b-kserve-64c8f55b49-bz2k8\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") " pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:46.931282 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:46.931218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:47.203520 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:47.203493    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"]
Sep 17 23:25:47.205700 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:25:47.205677    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode473c71a_5795_4b39_9737_0d92441c33ee.slice/crio-593bb713a967834f2bfb43e2de77793823ae7e5eaa900d762455b69267286439 WatchSource:0}: Error finding container 593bb713a967834f2bfb43e2de77793823ae7e5eaa900d762455b69267286439: Status 404 returned error can't find the container with id 593bb713a967834f2bfb43e2de77793823ae7e5eaa900d762455b69267286439
Sep 17 23:25:48.080993 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:48.080960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8" event={"ID":"e473c71a-5795-4b39-9737-0d92441c33ee","Type":"ContainerStarted","Data":"b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e"}
Sep 17 23:25:48.081377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:48.081001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8" event={"ID":"e473c71a-5795-4b39-9737-0d92441c33ee","Type":"ContainerStarted","Data":"593bb713a967834f2bfb43e2de77793823ae7e5eaa900d762455b69267286439"}
Sep 17 23:25:48.099553 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:48.099506    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8" podStartSLOduration=2.0994907 podStartE2EDuration="2.0994907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:48.098310559 +0000 UTC m=+1190.572253544" watchObservedRunningTime="2026-09-17 23:25:48.0994907 +0000 UTC m=+1190.573433727"
Sep 17 23:25:50.740462 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:50.740424    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:25:51.475339 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.475309    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"]
Sep 17 23:25:51.478967 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.478952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.481372 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.481349    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7f7059-dockercfg-nkbjn\""
Sep 17 23:25:51.481682 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.481668    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7f7059\""
Sep 17 23:25:51.491873 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.491850    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"]
Sep 17 23:25:51.533018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.532993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhn9\" (UniqueName: \"kubernetes.io/projected/4267b8ba-a813-4721-a243-6d15b415dd99-kube-api-access-mkhn9\") pod \"payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.533156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.533098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4267b8ba-a813-4721-a243-6d15b415dd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.549063 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.549032    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"]
Sep 17 23:25:51.552420 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.552403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.567265 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.567245    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"]
Sep 17 23:25:51.634108 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.634082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhn9\" (UniqueName: \"kubernetes.io/projected/4267b8ba-a813-4721-a243-6d15b415dd99-kube-api-access-mkhn9\") pod \"payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.634237 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.634147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4267b8ba-a813-4721-a243-6d15b415dd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.634237 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.634175    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1452ad00-5c75-4f4f-8929-72271d423a7a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") " pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.634237 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.634197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrf45\" (UniqueName: \"kubernetes.io/projected/1452ad00-5c75-4f4f-8929-72271d423a7a-kube-api-access-mrf45\") pod \"payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") " pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.634878 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.634859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4267b8ba-a813-4721-a243-6d15b415dd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.642561 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.642538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhn9\" (UniqueName: \"kubernetes.io/projected/4267b8ba-a813-4721-a243-6d15b415dd99-kube-api-access-mkhn9\") pod \"payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.735118 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.735070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrf45\" (UniqueName: \"kubernetes.io/projected/1452ad00-5c75-4f4f-8929-72271d423a7a-kube-api-access-mrf45\") pod \"payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") " pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.735208 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.735152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1452ad00-5c75-4f4f-8929-72271d423a7a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") " pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.735704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.735686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1452ad00-5c75-4f4f-8929-72271d423a7a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") " pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.744300 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.744281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrf45\" (UniqueName: \"kubernetes.io/projected/1452ad00-5c75-4f4f-8929-72271d423a7a-kube-api-access-mrf45\") pod \"payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") " pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:51.789778 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.789752    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:51.862051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:51.862021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:52.060704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:52.060676    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"]
Sep 17 23:25:52.062164 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:25:52.062135    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4267b8ba_a813_4721_a243_6d15b415dd99.slice/crio-b40e1eb2df4a9049482c4cb4f599a85ec58e7bda83086a2cdab45be474937084 WatchSource:0}: Error finding container b40e1eb2df4a9049482c4cb4f599a85ec58e7bda83086a2cdab45be474937084: Status 404 returned error can't find the container with id b40e1eb2df4a9049482c4cb4f599a85ec58e7bda83086a2cdab45be474937084
Sep 17 23:25:52.104397 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:52.104373    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx" event={"ID":"4267b8ba-a813-4721-a243-6d15b415dd99","Type":"ContainerStarted","Data":"b40e1eb2df4a9049482c4cb4f599a85ec58e7bda83086a2cdab45be474937084"}
Sep 17 23:25:52.134416 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:52.134394    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"]
Sep 17 23:25:52.136810 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:25:52.136775    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1452ad00_5c75_4f4f_8929_72271d423a7a.slice/crio-bd81c8f1359d627d0bd02f4761a01ec75720f9e7ebd1350882515277999b929c WatchSource:0}: Error finding container bd81c8f1359d627d0bd02f4761a01ec75720f9e7ebd1350882515277999b929c: Status 404 returned error can't find the container with id bd81c8f1359d627d0bd02f4761a01ec75720f9e7ebd1350882515277999b929c
Sep 17 23:25:53.111235 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.111196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl" event={"ID":"1452ad00-5c75-4f4f-8929-72271d423a7a","Type":"ContainerStarted","Data":"113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5"}
Sep 17 23:25:53.111858 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.111386    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:25:53.111858 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.111412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl" event={"ID":"1452ad00-5c75-4f4f-8929-72271d423a7a","Type":"ContainerStarted","Data":"bd81c8f1359d627d0bd02f4761a01ec75720f9e7ebd1350882515277999b929c"}
Sep 17 23:25:53.112692 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.112660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx" event={"ID":"4267b8ba-a813-4721-a243-6d15b415dd99","Type":"ContainerStarted","Data":"705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398"}
Sep 17 23:25:53.112826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.112749    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:25:53.143249 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.143207    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl" podStartSLOduration=2.143191941 podStartE2EDuration="2.143191941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:53.14174774 +0000 UTC m=+1195.615690723" watchObservedRunningTime="2026-09-17 23:25:53.143191941 +0000 UTC m=+1195.617134927"
Sep 17 23:25:53.173965 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:53.173928    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx" podStartSLOduration=2.173915616 podStartE2EDuration="2.173915616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:53.171401161 +0000 UTC m=+1195.645344143" watchObservedRunningTime="2026-09-17 23:25:53.173915616 +0000 UTC m=+1195.647858599"
Sep 17 23:25:56.932033 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:56.931997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:56.932511 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:56.932077    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:56.944833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:56.944811    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:57.147259 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:57.147239    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:25:59.627410 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:59.627384    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:25:59.629299 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:59.629276    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:25:59.639315 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:59.639293    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:25:59.641744 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:25:59.641710    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:26:04.121441 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:04.121403    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:26:04.121871 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:04.121474    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:26:10.438157 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:10.438120    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:26:14.100401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.100365    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"]
Sep 17 23:26:14.105552 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.105535    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.107977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.107944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df969c31-dockercfg-pfht4\""
Sep 17 23:26:14.107977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.107970    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df969c31\""
Sep 17 23:26:14.139752 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.139712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"]
Sep 17 23:26:14.222673 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.222651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.222790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.222744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxccc\" (UniqueName: \"kubernetes.io/projected/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-kube-api-access-dxccc\") pod \"payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.323635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.323615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxccc\" (UniqueName: \"kubernetes.io/projected/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-kube-api-access-dxccc\") pod \"payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.323755 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.323658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.324273 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.324255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.332134 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.332106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxccc\" (UniqueName: \"kubernetes.io/projected/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-kube-api-access-dxccc\") pod \"payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.415434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.415381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:14.687252 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:14.687227    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"]
Sep 17 23:26:14.688911 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:26:14.688886    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52e8a7f_3628_497e_96c0_1c19aaca8a8d.slice/crio-0a4a548eb30bb4e401f6d8274a7177d4feaa2f524e86768aa56e142fb84bb251 WatchSource:0}: Error finding container 0a4a548eb30bb4e401f6d8274a7177d4feaa2f524e86768aa56e142fb84bb251: Status 404 returned error can't find the container with id 0a4a548eb30bb4e401f6d8274a7177d4feaa2f524e86768aa56e142fb84bb251
Sep 17 23:26:15.236621 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:15.236586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x" event={"ID":"d52e8a7f-3628-497e-96c0-1c19aaca8a8d","Type":"ContainerStarted","Data":"8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9"}
Sep 17 23:26:15.237039 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:15.236691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x" event={"ID":"d52e8a7f-3628-497e-96c0-1c19aaca8a8d","Type":"ContainerStarted","Data":"0a4a548eb30bb4e401f6d8274a7177d4feaa2f524e86768aa56e142fb84bb251"}
Sep 17 23:26:15.237039 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:15.236756    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:15.253972 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:15.253926    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x" podStartSLOduration=1.253907473 podStartE2EDuration="1.253907473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:15.251619272 +0000 UTC m=+1217.725562272" watchObservedRunningTime="2026-09-17 23:26:15.253907473 +0000 UTC m=+1217.727850457"
Sep 17 23:26:21.935890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:21.935850    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:26:26.244142 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:26.244109    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:46.397939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.397857    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"]
Sep 17 23:26:46.398701 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.398667    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl" podUID="1452ad00-5c75-4f4f-8929-72271d423a7a" containerName="payload-processing" containerID="cri-o://113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5" gracePeriod=30
Sep 17 23:26:46.402046 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.402013    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"]
Sep 17 23:26:46.402258 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.402211    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx" podUID="4267b8ba-a813-4721-a243-6d15b415dd99" containerName="payload-pre-processing" containerID="cri-o://705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398" gracePeriod=30
Sep 17 23:26:46.664060 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.664004    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:26:46.679868 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.679845    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:26:46.801853 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.801825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrf45\" (UniqueName: \"kubernetes.io/projected/1452ad00-5c75-4f4f-8929-72271d423a7a-kube-api-access-mrf45\") pod \"1452ad00-5c75-4f4f-8929-72271d423a7a\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") "
Sep 17 23:26:46.802018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.801866    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1452ad00-5c75-4f4f-8929-72271d423a7a-plugins-config-volume\") pod \"1452ad00-5c75-4f4f-8929-72271d423a7a\" (UID: \"1452ad00-5c75-4f4f-8929-72271d423a7a\") "
Sep 17 23:26:46.802018 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.801914    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkhn9\" (UniqueName: \"kubernetes.io/projected/4267b8ba-a813-4721-a243-6d15b415dd99-kube-api-access-mkhn9\") pod \"4267b8ba-a813-4721-a243-6d15b415dd99\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") "
Sep 17 23:26:46.802137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.802026    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4267b8ba-a813-4721-a243-6d15b415dd99-plugins-config-volume\") pod \"4267b8ba-a813-4721-a243-6d15b415dd99\" (UID: \"4267b8ba-a813-4721-a243-6d15b415dd99\") "
Sep 17 23:26:46.802345 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.802320    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1452ad00-5c75-4f4f-8929-72271d423a7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1452ad00-5c75-4f4f-8929-72271d423a7a" (UID: "1452ad00-5c75-4f4f-8929-72271d423a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:46.802412 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.802390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4267b8ba-a813-4721-a243-6d15b415dd99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4267b8ba-a813-4721-a243-6d15b415dd99" (UID: "4267b8ba-a813-4721-a243-6d15b415dd99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:46.803933 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.803912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1452ad00-5c75-4f4f-8929-72271d423a7a-kube-api-access-mrf45" (OuterVolumeSpecName: "kube-api-access-mrf45") pod "1452ad00-5c75-4f4f-8929-72271d423a7a" (UID: "1452ad00-5c75-4f4f-8929-72271d423a7a"). InnerVolumeSpecName "kube-api-access-mrf45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:46.803995 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.803941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4267b8ba-a813-4721-a243-6d15b415dd99-kube-api-access-mkhn9" (OuterVolumeSpecName: "kube-api-access-mkhn9") pod "4267b8ba-a813-4721-a243-6d15b415dd99" (UID: "4267b8ba-a813-4721-a243-6d15b415dd99"). InnerVolumeSpecName "kube-api-access-mkhn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:46.903678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.903650    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhn9\" (UniqueName: \"kubernetes.io/projected/4267b8ba-a813-4721-a243-6d15b415dd99-kube-api-access-mkhn9\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:46.903678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.903673    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4267b8ba-a813-4721-a243-6d15b415dd99-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:46.903678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.903683    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrf45\" (UniqueName: \"kubernetes.io/projected/1452ad00-5c75-4f4f-8929-72271d423a7a-kube-api-access-mrf45\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:46.903882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:46.903709    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1452ad00-5c75-4f4f-8929-72271d423a7a-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:47.302999 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.302961    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"]
Sep 17 23:26:47.303537 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.303521    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1452ad00-5c75-4f4f-8929-72271d423a7a" containerName="payload-processing"
Sep 17 23:26:47.303583 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.303541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1452ad00-5c75-4f4f-8929-72271d423a7a" containerName="payload-processing"
Sep 17 23:26:47.303583 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.303556    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4267b8ba-a813-4721-a243-6d15b415dd99" containerName="payload-pre-processing"
Sep 17 23:26:47.303583 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.303564    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4267b8ba-a813-4721-a243-6d15b415dd99" containerName="payload-pre-processing"
Sep 17 23:26:47.303681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.303661    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4267b8ba-a813-4721-a243-6d15b415dd99" containerName="payload-pre-processing"
Sep 17 23:26:47.303681 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.303672    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1452ad00-5c75-4f4f-8929-72271d423a7a" containerName="payload-processing"
Sep 17 23:26:47.307274 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.307253    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:47.309272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.309253    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:26:47.309368 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.309303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\""
Sep 17 23:26:47.313635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.313608    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"]
Sep 17 23:26:47.408205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.408172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df969c31-w6f5w\" (UID: \"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:47.423055 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.423023    2559 generic.go:356] "Generic (PLEG): container finished" podID="1452ad00-5c75-4f4f-8929-72271d423a7a" containerID="113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5" exitCode=0
Sep 17 23:26:47.423170 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.423069    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"
Sep 17 23:26:47.423170 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.423108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl" event={"ID":"1452ad00-5c75-4f4f-8929-72271d423a7a","Type":"ContainerDied","Data":"113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5"}
Sep 17 23:26:47.423170 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.423158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl" event={"ID":"1452ad00-5c75-4f4f-8929-72271d423a7a","Type":"ContainerDied","Data":"bd81c8f1359d627d0bd02f4761a01ec75720f9e7ebd1350882515277999b929c"}
Sep 17 23:26:47.423334 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.423192    2559 scope.go:117] "RemoveContainer" containerID="113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5"
Sep 17 23:26:47.424800 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.424778    2559 generic.go:356] "Generic (PLEG): container finished" podID="4267b8ba-a813-4721-a243-6d15b415dd99" containerID="705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398" exitCode=0
Sep 17 23:26:47.424901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.424839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx" event={"ID":"4267b8ba-a813-4721-a243-6d15b415dd99","Type":"ContainerDied","Data":"705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398"}
Sep 17 23:26:47.424901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.424875    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"
Sep 17 23:26:47.424901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.424888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx" event={"ID":"4267b8ba-a813-4721-a243-6d15b415dd99","Type":"ContainerDied","Data":"b40e1eb2df4a9049482c4cb4f599a85ec58e7bda83086a2cdab45be474937084"}
Sep 17 23:26:47.432912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.432896    2559 scope.go:117] "RemoveContainer" containerID="113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5"
Sep 17 23:26:47.433190 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:26:47.433171    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5\": container with ID starting with 113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5 not found: ID does not exist" containerID="113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5"
Sep 17 23:26:47.433264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.433198    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5"} err="failed to get container status \"113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5\": rpc error: code = NotFound desc = could not find container \"113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5\": container with ID starting with 113f8f67f205fb8fce8e4daa4d717e14b4e8e94e09923bc5ae62f6bc6ffe27a5 not found: ID does not exist"
Sep 17 23:26:47.433264 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.433215    2559 scope.go:117] "RemoveContainer" containerID="705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398"
Sep 17 23:26:47.442356 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.442339    2559 scope.go:117] "RemoveContainer" containerID="705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398"
Sep 17 23:26:47.442665 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:26:47.442642    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398\": container with ID starting with 705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398 not found: ID does not exist" containerID="705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398"
Sep 17 23:26:47.442753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.442675    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398"} err="failed to get container status \"705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398\": rpc error: code = NotFound desc = could not find container \"705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398\": container with ID starting with 705b056408e376f218a9245cb81fab7b70e0df5dd67126ebd3c7270bd0681398 not found: ID does not exist"
Sep 17 23:26:47.449901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.449882    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"]
Sep 17 23:26:47.451878 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.451861    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7f7059-c8fdb9cb5-24bkl"]
Sep 17 23:26:47.460552 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.460532    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"]
Sep 17 23:26:47.462797 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.462779    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7f7059-59b974686b-66hrx"]
Sep 17 23:26:47.509690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.509669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df969c31-w6f5w\" (UID: \"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:47.510227 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.510209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df969c31-w6f5w\" (UID: \"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:47.619013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.618946    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:47.640585 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.640558    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1452ad00-5c75-4f4f-8929-72271d423a7a" path="/var/lib/kubelet/pods/1452ad00-5c75-4f4f-8929-72271d423a7a/volumes"
Sep 17 23:26:47.640955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.640938    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4267b8ba-a813-4721-a243-6d15b415dd99" path="/var/lib/kubelet/pods/4267b8ba-a813-4721-a243-6d15b415dd99/volumes"
Sep 17 23:26:47.890577 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:47.890498    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"]
Sep 17 23:26:47.893934 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:26:47.893910    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3e5cd0_bf2a_4bfd_a777_2e79880c8897.slice/crio-d432c6332fd5c89475b0364558e008dfbfa550bc43df8a022e1334e32ce0dcd5 WatchSource:0}: Error finding container d432c6332fd5c89475b0364558e008dfbfa550bc43df8a022e1334e32ce0dcd5: Status 404 returned error can't find the container with id d432c6332fd5c89475b0364558e008dfbfa550bc43df8a022e1334e32ce0dcd5
Sep 17 23:26:48.433066 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:48.433027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w" event={"ID":"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897","Type":"ContainerStarted","Data":"8c93561718574abcc60facd2d6301ba2779e5bc8f3a3b4a5176ca62ceb562585"}
Sep 17 23:26:48.433429 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:48.433087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w" event={"ID":"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897","Type":"ContainerStarted","Data":"d432c6332fd5c89475b0364558e008dfbfa550bc43df8a022e1334e32ce0dcd5"}
Sep 17 23:26:48.450071 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:48.450029    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w" podStartSLOduration=1.450013969 podStartE2EDuration="1.450013969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:48.448643028 +0000 UTC m=+1250.922586010" watchObservedRunningTime="2026-09-17 23:26:48.450013969 +0000 UTC m=+1250.923956945"
Sep 17 23:26:49.441165 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:49.441131    2559 generic.go:356] "Generic (PLEG): container finished" podID="5c3e5cd0-bf2a-4bfd-a777-2e79880c8897" containerID="8c93561718574abcc60facd2d6301ba2779e5bc8f3a3b4a5176ca62ceb562585" exitCode=0
Sep 17 23:26:49.441560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:49.441177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w" event={"ID":"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897","Type":"ContainerDied","Data":"8c93561718574abcc60facd2d6301ba2779e5bc8f3a3b4a5176ca62ceb562585"}
Sep 17 23:26:50.579552 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:50.579529    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:50.636810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:50.636782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897-maas-api-service-ca\") pod \"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897\" (UID: \"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897\") "
Sep 17 23:26:50.637097 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:50.637077    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c3e5cd0-bf2a-4bfd-a777-2e79880c8897" (UID: "5c3e5cd0-bf2a-4bfd-a777-2e79880c8897"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:50.738390 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:50.738354    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897-maas-api-service-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:50.934006 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:50.933967    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:26:51.455037 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:51.455002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w" event={"ID":"5c3e5cd0-bf2a-4bfd-a777-2e79880c8897","Type":"ContainerDied","Data":"d432c6332fd5c89475b0364558e008dfbfa550bc43df8a022e1334e32ce0dcd5"}
Sep 17 23:26:51.455202 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:51.455060    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d432c6332fd5c89475b0364558e008dfbfa550bc43df8a022e1334e32ce0dcd5"
Sep 17 23:26:51.455202 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:51.455009    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"
Sep 17 23:26:55.619628 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:55.619595    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"]
Sep 17 23:26:57.006637 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.006596    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"]
Sep 17 23:26:57.127772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.127743    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"]
Sep 17 23:26:57.128213 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.128195    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3e5cd0-bf2a-4bfd-a777-2e79880c8897" containerName="revoke-keys"
Sep 17 23:26:57.128360 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.128216    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3e5cd0-bf2a-4bfd-a777-2e79880c8897" containerName="revoke-keys"
Sep 17 23:26:57.128360 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.128316    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3e5cd0-bf2a-4bfd-a777-2e79880c8897" containerName="revoke-keys"
Sep 17 23:26:57.132013 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.131991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:26:57.134303 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.134277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\""
Sep 17 23:26:57.134398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.134288    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:26:57.138261 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.138239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"]
Sep 17 23:26:57.296620 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.296544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b59e-fd9c-4a82-970d-fa07595f2108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl\" (UID: \"1877b59e-fd9c-4a82-970d-fa07595f2108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:26:57.397043 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.397019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b59e-fd9c-4a82-970d-fa07595f2108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl\" (UID: \"1877b59e-fd9c-4a82-970d-fa07595f2108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:26:57.397582 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.397561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b59e-fd9c-4a82-970d-fa07595f2108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl\" (UID: \"1877b59e-fd9c-4a82-970d-fa07595f2108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:26:57.446070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.446048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:26:57.724885 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:57.724843    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"]
Sep 17 23:26:57.727391 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:26:57.727364    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1877b59e_fd9c_4a82_970d_fa07595f2108.slice/crio-eaaf099f0ab16182c0bb6b3e6374942c1a0c2469a2353d440501904b16f7b8ae WatchSource:0}: Error finding container eaaf099f0ab16182c0bb6b3e6374942c1a0c2469a2353d440501904b16f7b8ae: Status 404 returned error can't find the container with id eaaf099f0ab16182c0bb6b3e6374942c1a0c2469a2353d440501904b16f7b8ae
Sep 17 23:26:58.148251 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.148220    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"]
Sep 17 23:26:58.148690 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.148412    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x" podUID="d52e8a7f-3628-497e-96c0-1c19aaca8a8d" containerName="payload-pre-processing" containerID="cri-o://8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9" gracePeriod=30
Sep 17 23:26:58.312839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.312766    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"]
Sep 17 23:26:58.316024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.315999    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df969c31-w6f5w"]
Sep 17 23:26:58.382550 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.382527    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:58.494972 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.494943    2559 generic.go:356] "Generic (PLEG): container finished" podID="d52e8a7f-3628-497e-96c0-1c19aaca8a8d" containerID="8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9" exitCode=0
Sep 17 23:26:58.495083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.494990    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"
Sep 17 23:26:58.495083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.495052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x" event={"ID":"d52e8a7f-3628-497e-96c0-1c19aaca8a8d","Type":"ContainerDied","Data":"8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9"}
Sep 17 23:26:58.495196 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.495103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x" event={"ID":"d52e8a7f-3628-497e-96c0-1c19aaca8a8d","Type":"ContainerDied","Data":"0a4a548eb30bb4e401f6d8274a7177d4feaa2f524e86768aa56e142fb84bb251"}
Sep 17 23:26:58.495196 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.495133    2559 scope.go:117] "RemoveContainer" containerID="8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9"
Sep 17 23:26:58.496765 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.496745    2559 generic.go:356] "Generic (PLEG): container finished" podID="1877b59e-fd9c-4a82-970d-fa07595f2108" containerID="13c45e85e8ed9c9232ea286a1c86d132ca99a81f7d03231cf0adab6223449f70" exitCode=0
Sep 17 23:26:58.496853 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.496781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl" event={"ID":"1877b59e-fd9c-4a82-970d-fa07595f2108","Type":"ContainerDied","Data":"13c45e85e8ed9c9232ea286a1c86d132ca99a81f7d03231cf0adab6223449f70"}
Sep 17 23:26:58.496853 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.496816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl" event={"ID":"1877b59e-fd9c-4a82-970d-fa07595f2108","Type":"ContainerStarted","Data":"eaaf099f0ab16182c0bb6b3e6374942c1a0c2469a2353d440501904b16f7b8ae"}
Sep 17 23:26:58.503564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.503548    2559 scope.go:117] "RemoveContainer" containerID="8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9"
Sep 17 23:26:58.503920 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:26:58.503900    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9\": container with ID starting with 8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9 not found: ID does not exist" containerID="8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9"
Sep 17 23:26:58.503985 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.503931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9"} err="failed to get container status \"8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9\": rpc error: code = NotFound desc = could not find container \"8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9\": container with ID starting with 8c44f3875eeaae5a972e6bd32a509efbe5dce69d7ba2c92ac33ed5a6762b89f9 not found: ID does not exist"
Sep 17 23:26:58.508741 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.508704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxccc\" (UniqueName: \"kubernetes.io/projected/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-kube-api-access-dxccc\") pod \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") "
Sep 17 23:26:58.508815 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.508758    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-plugins-config-volume\") pod \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\" (UID: \"d52e8a7f-3628-497e-96c0-1c19aaca8a8d\") "
Sep 17 23:26:58.509135 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.509117    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d52e8a7f-3628-497e-96c0-1c19aaca8a8d" (UID: "d52e8a7f-3628-497e-96c0-1c19aaca8a8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:58.510628 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.510604    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-kube-api-access-dxccc" (OuterVolumeSpecName: "kube-api-access-dxccc") pod "d52e8a7f-3628-497e-96c0-1c19aaca8a8d" (UID: "d52e8a7f-3628-497e-96c0-1c19aaca8a8d"). InnerVolumeSpecName "kube-api-access-dxccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:58.609735 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.609667    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxccc\" (UniqueName: \"kubernetes.io/projected/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-kube-api-access-dxccc\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:58.609735 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.609687    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52e8a7f-3628-497e-96c0-1c19aaca8a8d-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:26:58.815558 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.815525    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"]
Sep 17 23:26:58.819854 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:58.819828    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df969c31-6d66f77474-7r94x"]
Sep 17 23:26:59.641907 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:59.641879    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3e5cd0-bf2a-4bfd-a777-2e79880c8897" path="/var/lib/kubelet/pods/5c3e5cd0-bf2a-4bfd-a777-2e79880c8897/volumes"
Sep 17 23:26:59.642278 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:59.642263    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52e8a7f-3628-497e-96c0-1c19aaca8a8d" path="/var/lib/kubelet/pods/d52e8a7f-3628-497e-96c0-1c19aaca8a8d/volumes"
Sep 17 23:26:59.643048 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:59.643030    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:26:59.718762 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:59.718741    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b59e-fd9c-4a82-970d-fa07595f2108-maas-api-service-ca\") pod \"1877b59e-fd9c-4a82-970d-fa07595f2108\" (UID: \"1877b59e-fd9c-4a82-970d-fa07595f2108\") "
Sep 17 23:26:59.718960 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:59.718938    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1877b59e-fd9c-4a82-970d-fa07595f2108-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1877b59e-fd9c-4a82-970d-fa07595f2108" (UID: "1877b59e-fd9c-4a82-970d-fa07595f2108"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:59.719408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:26:59.719379    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b59e-fd9c-4a82-970d-fa07595f2108-maas-api-service-ca\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:00.509774 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:00.509714    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"
Sep 17 23:27:00.509979 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:00.509713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl" event={"ID":"1877b59e-fd9c-4a82-970d-fa07595f2108","Type":"ContainerDied","Data":"eaaf099f0ab16182c0bb6b3e6374942c1a0c2469a2353d440501904b16f7b8ae"}
Sep 17 23:27:00.509979 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:00.509870    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaaf099f0ab16182c0bb6b3e6374942c1a0c2469a2353d440501904b16f7b8ae"
Sep 17 23:27:04.724784 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:04.724749    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:27:07.891357 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:07.891309    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"]
Sep 17 23:27:07.891825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:07.891529    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd" podUID="40446d60-31db-403b-a8a2-a1b3f908ac02" containerName="payload-pre-processing" containerID="cri-o://8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47" gracePeriod=30
Sep 17 23:27:08.096202 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.096171    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"]
Sep 17 23:27:08.121452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.121422    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c363f-bm8jl"]
Sep 17 23:27:08.164381 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.164355    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:27:08.299531 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.299499    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40446d60-31db-403b-a8a2-a1b3f908ac02-plugins-config-volume\") pod \"40446d60-31db-403b-a8a2-a1b3f908ac02\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") "
Sep 17 23:27:08.299761 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.299622    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zstqg\" (UniqueName: \"kubernetes.io/projected/40446d60-31db-403b-a8a2-a1b3f908ac02-kube-api-access-zstqg\") pod \"40446d60-31db-403b-a8a2-a1b3f908ac02\" (UID: \"40446d60-31db-403b-a8a2-a1b3f908ac02\") "
Sep 17 23:27:08.300010 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.299983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40446d60-31db-403b-a8a2-a1b3f908ac02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40446d60-31db-403b-a8a2-a1b3f908ac02" (UID: "40446d60-31db-403b-a8a2-a1b3f908ac02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:08.301830 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.301804    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40446d60-31db-403b-a8a2-a1b3f908ac02-kube-api-access-zstqg" (OuterVolumeSpecName: "kube-api-access-zstqg") pod "40446d60-31db-403b-a8a2-a1b3f908ac02" (UID: "40446d60-31db-403b-a8a2-a1b3f908ac02"). InnerVolumeSpecName "kube-api-access-zstqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:08.400734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.400629    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40446d60-31db-403b-a8a2-a1b3f908ac02-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:08.400734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.400662    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstqg\" (UniqueName: \"kubernetes.io/projected/40446d60-31db-403b-a8a2-a1b3f908ac02-kube-api-access-zstqg\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:08.555565 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.555532    2559 generic.go:356] "Generic (PLEG): container finished" podID="40446d60-31db-403b-a8a2-a1b3f908ac02" containerID="8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47" exitCode=0
Sep 17 23:27:08.555750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.555574    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"
Sep 17 23:27:08.555750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.555602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd" event={"ID":"40446d60-31db-403b-a8a2-a1b3f908ac02","Type":"ContainerDied","Data":"8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47"}
Sep 17 23:27:08.555750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.555642    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd" event={"ID":"40446d60-31db-403b-a8a2-a1b3f908ac02","Type":"ContainerDied","Data":"2effe9cebf3346fc3ad08cd73dde92e8c075719814fb38961af6c08a7a2c9c09"}
Sep 17 23:27:08.555750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.555662    2559 scope.go:117] "RemoveContainer" containerID="8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47"
Sep 17 23:27:08.564601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.564585    2559 scope.go:117] "RemoveContainer" containerID="8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47"
Sep 17 23:27:08.564934 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:27:08.564913    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47\": container with ID starting with 8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47 not found: ID does not exist" containerID="8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47"
Sep 17 23:27:08.564997 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.564943    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47"} err="failed to get container status \"8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47\": rpc error: code = NotFound desc = could not find container \"8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47\": container with ID starting with 8dc7bd91e8c90ae60667a3eedd8dc5334f441a7922e983a6cec74aa617f07e47 not found: ID does not exist"
Sep 17 23:27:08.576321 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.576300    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"]
Sep 17 23:27:08.582117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:08.582095    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c363f-7c69fd5846-pjkvd"]
Sep 17 23:27:09.640698 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:09.640669    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1877b59e-fd9c-4a82-970d-fa07595f2108" path="/var/lib/kubelet/pods/1877b59e-fd9c-4a82-970d-fa07595f2108/volumes"
Sep 17 23:27:09.641084 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:09.641070    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40446d60-31db-403b-a8a2-a1b3f908ac02" path="/var/lib/kubelet/pods/40446d60-31db-403b-a8a2-a1b3f908ac02/volumes"
Sep 17 23:27:10.639523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.639477    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756" podUID="70ef574f-b577-4145-b3c7-ec1f335d5ed2" containerName="main" containerID="cri-o://be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48" gracePeriod=45
Sep 17 23:27:10.882104 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.882081    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:27:10.923864 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.923799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-home\") pod \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") "
Sep 17 23:27:10.923864 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.923853    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tmp-dir\") pod \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") "
Sep 17 23:27:10.924051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.923877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp4nd\" (UniqueName: \"kubernetes.io/projected/70ef574f-b577-4145-b3c7-ec1f335d5ed2-kube-api-access-tp4nd\") pod \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") "
Sep 17 23:27:10.924051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.923897    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tls-certs\") pod \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") "
Sep 17 23:27:10.924051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-dshm\") pod \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") "
Sep 17 23:27:10.924203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924058    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70ef574f-b577-4145-b3c7-ec1f335d5ed2" (UID: "70ef574f-b577-4145-b3c7-ec1f335d5ed2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:10.924203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-model-cache\") pod \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\" (UID: \"70ef574f-b577-4145-b3c7-ec1f335d5ed2\") "
Sep 17 23:27:10.924203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-home" (OuterVolumeSpecName: "home") pod "70ef574f-b577-4145-b3c7-ec1f335d5ed2" (UID: "70ef574f-b577-4145-b3c7-ec1f335d5ed2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:10.924353 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-model-cache" (OuterVolumeSpecName: "model-cache") pod "70ef574f-b577-4145-b3c7-ec1f335d5ed2" (UID: "70ef574f-b577-4145-b3c7-ec1f335d5ed2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:10.924409 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924399    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.924464 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924412    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.924464 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.924422    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.926187 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.926159    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-dshm" (OuterVolumeSpecName: "dshm") pod "70ef574f-b577-4145-b3c7-ec1f335d5ed2" (UID: "70ef574f-b577-4145-b3c7-ec1f335d5ed2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:10.926277 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.926166    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70ef574f-b577-4145-b3c7-ec1f335d5ed2" (UID: "70ef574f-b577-4145-b3c7-ec1f335d5ed2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:10.926430 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:10.926416    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ef574f-b577-4145-b3c7-ec1f335d5ed2-kube-api-access-tp4nd" (OuterVolumeSpecName: "kube-api-access-tp4nd") pod "70ef574f-b577-4145-b3c7-ec1f335d5ed2" (UID: "70ef574f-b577-4145-b3c7-ec1f335d5ed2"). InnerVolumeSpecName "kube-api-access-tp4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:11.024907 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.024883    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4nd\" (UniqueName: \"kubernetes.io/projected/70ef574f-b577-4145-b3c7-ec1f335d5ed2-kube-api-access-tp4nd\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:11.024907 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.024905    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef574f-b577-4145-b3c7-ec1f335d5ed2-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:11.025043 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.024915    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef574f-b577-4145-b3c7-ec1f335d5ed2-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:11.573430 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.573397    2559 generic.go:356] "Generic (PLEG): container finished" podID="70ef574f-b577-4145-b3c7-ec1f335d5ed2" containerID="be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48" exitCode=0
Sep 17 23:27:11.573613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.573437    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"
Sep 17 23:27:11.573613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.573488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756" event={"ID":"70ef574f-b577-4145-b3c7-ec1f335d5ed2","Type":"ContainerDied","Data":"be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48"}
Sep 17 23:27:11.573613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.573538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756" event={"ID":"70ef574f-b577-4145-b3c7-ec1f335d5ed2","Type":"ContainerDied","Data":"365063cb2af49102da002fca4efd732e36f494614608c4ff9299dd565fba5cf5"}
Sep 17 23:27:11.573613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.573564    2559 scope.go:117] "RemoveContainer" containerID="be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48"
Sep 17 23:27:11.583005 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.582983    2559 scope.go:117] "RemoveContainer" containerID="be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48"
Sep 17 23:27:11.583287 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:27:11.583266    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48\": container with ID starting with be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48 not found: ID does not exist" containerID="be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48"
Sep 17 23:27:11.583333 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.583296    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48"} err="failed to get container status \"be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48\": rpc error: code = NotFound desc = could not find container \"be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48\": container with ID starting with be4a9a140d0e336dc24697a612579bc1d6e6f1cdc3636f6055a88acbce118a48 not found: ID does not exist"
Sep 17 23:27:11.594499 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.594467    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"]
Sep 17 23:27:11.600451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.600426    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0c363f/test-model-0c363f-kserve-7d79f55cf5-nr756"]
Sep 17 23:27:11.639373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:11.639341    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ef574f-b577-4145-b3c7-ec1f335d5ed2" path="/var/lib/kubelet/pods/70ef574f-b577-4145-b3c7-ec1f335d5ed2/volumes"
Sep 17 23:27:12.026568 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.026528    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8" podUID="e473c71a-5795-4b39-9737-0d92441c33ee" containerName="main" containerID="cri-o://b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e" gracePeriod=45
Sep 17 23:27:12.271022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.270997    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:27:12.337087 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337008    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-dshm\") pod \"e473c71a-5795-4b39-9737-0d92441c33ee\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") "
Sep 17 23:27:12.337087 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-tmp-dir\") pod \"e473c71a-5795-4b39-9737-0d92441c33ee\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") "
Sep 17 23:27:12.337087 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337088    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr9rp\" (UniqueName: \"kubernetes.io/projected/e473c71a-5795-4b39-9737-0d92441c33ee-kube-api-access-tr9rp\") pod \"e473c71a-5795-4b39-9737-0d92441c33ee\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") "
Sep 17 23:27:12.337367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337105    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e473c71a-5795-4b39-9737-0d92441c33ee-tls-certs\") pod \"e473c71a-5795-4b39-9737-0d92441c33ee\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") "
Sep 17 23:27:12.337367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-home\") pod \"e473c71a-5795-4b39-9737-0d92441c33ee\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") "
Sep 17 23:27:12.337367 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-model-cache\") pod \"e473c71a-5795-4b39-9737-0d92441c33ee\" (UID: \"e473c71a-5795-4b39-9737-0d92441c33ee\") "
Sep 17 23:27:12.337524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337425    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-home" (OuterVolumeSpecName: "home") pod "e473c71a-5795-4b39-9737-0d92441c33ee" (UID: "e473c71a-5795-4b39-9737-0d92441c33ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:12.337524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e473c71a-5795-4b39-9737-0d92441c33ee" (UID: "e473c71a-5795-4b39-9737-0d92441c33ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:12.337524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.337490    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "e473c71a-5795-4b39-9737-0d92441c33ee" (UID: "e473c71a-5795-4b39-9737-0d92441c33ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:12.339328 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.339303    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e473c71a-5795-4b39-9737-0d92441c33ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e473c71a-5795-4b39-9737-0d92441c33ee" (UID: "e473c71a-5795-4b39-9737-0d92441c33ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:12.339434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.339360    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-dshm" (OuterVolumeSpecName: "dshm") pod "e473c71a-5795-4b39-9737-0d92441c33ee" (UID: "e473c71a-5795-4b39-9737-0d92441c33ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:12.339434 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.339375    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473c71a-5795-4b39-9737-0d92441c33ee-kube-api-access-tr9rp" (OuterVolumeSpecName: "kube-api-access-tr9rp") pod "e473c71a-5795-4b39-9737-0d92441c33ee" (UID: "e473c71a-5795-4b39-9737-0d92441c33ee"). InnerVolumeSpecName "kube-api-access-tr9rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:12.438222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.438182    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:12.438222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.438214    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:12.438222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.438232    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr9rp\" (UniqueName: \"kubernetes.io/projected/e473c71a-5795-4b39-9737-0d92441c33ee-kube-api-access-tr9rp\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:12.438468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.438243    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e473c71a-5795-4b39-9737-0d92441c33ee-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:12.438468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.438254    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:12.438468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.438265    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e473c71a-5795-4b39-9737-0d92441c33ee-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:12.580999 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.580967    2559 generic.go:356] "Generic (PLEG): container finished" podID="e473c71a-5795-4b39-9737-0d92441c33ee" containerID="b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e" exitCode=0
Sep 17 23:27:12.581178 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.581008    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"
Sep 17 23:27:12.581178 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.581014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8" event={"ID":"e473c71a-5795-4b39-9737-0d92441c33ee","Type":"ContainerDied","Data":"b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e"}
Sep 17 23:27:12.581178 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.581044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8" event={"ID":"e473c71a-5795-4b39-9737-0d92441c33ee","Type":"ContainerDied","Data":"593bb713a967834f2bfb43e2de77793823ae7e5eaa900d762455b69267286439"}
Sep 17 23:27:12.581178 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.581060    2559 scope.go:117] "RemoveContainer" containerID="b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e"
Sep 17 23:27:12.590476 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.590458    2559 scope.go:117] "RemoveContainer" containerID="b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e"
Sep 17 23:27:12.590785 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:27:12.590766    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e\": container with ID starting with b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e not found: ID does not exist" containerID="b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e"
Sep 17 23:27:12.590847 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.590792    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e"} err="failed to get container status \"b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e\": rpc error: code = NotFound desc = could not find container \"b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e\": container with ID starting with b87b0c97619edfe233f37ec2622240caa8fa69881c55b1b190114ceefbaedf1e not found: ID does not exist"
Sep 17 23:27:12.601846 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.601821    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"]
Sep 17 23:27:12.605394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:12.605370    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-307a7b/test-model-307a7b-kserve-64c8f55b49-bz2k8"]
Sep 17 23:27:13.641403 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:13.641372    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e473c71a-5795-4b39-9737-0d92441c33ee" path="/var/lib/kubelet/pods/e473c71a-5795-4b39-9737-0d92441c33ee/volumes"
Sep 17 23:27:15.135298 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.135256    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:27:15.721468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721432    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"]
Sep 17 23:27:15.721887 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721869    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52e8a7f-3628-497e-96c0-1c19aaca8a8d" containerName="payload-pre-processing"
Sep 17 23:27:15.721887 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721887    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52e8a7f-3628-497e-96c0-1c19aaca8a8d" containerName="payload-pre-processing"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721904    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1877b59e-fd9c-4a82-970d-fa07595f2108" containerName="revoke-keys"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721911    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1877b59e-fd9c-4a82-970d-fa07595f2108" containerName="revoke-keys"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721928    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e473c71a-5795-4b39-9737-0d92441c33ee" containerName="main"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721936    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e473c71a-5795-4b39-9737-0d92441c33ee" containerName="main"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721947    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40446d60-31db-403b-a8a2-a1b3f908ac02" containerName="payload-pre-processing"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721956    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40446d60-31db-403b-a8a2-a1b3f908ac02" containerName="payload-pre-processing"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721972    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ef574f-b577-4145-b3c7-ec1f335d5ed2" containerName="main"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.721980    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ef574f-b577-4145-b3c7-ec1f335d5ed2" containerName="main"
Sep 17 23:27:15.722051 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.722055    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ef574f-b577-4145-b3c7-ec1f335d5ed2" containerName="main"
Sep 17 23:27:15.722369 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.722069    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1877b59e-fd9c-4a82-970d-fa07595f2108" containerName="revoke-keys"
Sep 17 23:27:15.722369 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.722081    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e473c71a-5795-4b39-9737-0d92441c33ee" containerName="main"
Sep 17 23:27:15.722369 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.722089    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40446d60-31db-403b-a8a2-a1b3f908ac02" containerName="payload-pre-processing"
Sep 17 23:27:15.722369 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.722096    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52e8a7f-3628-497e-96c0-1c19aaca8a8d" containerName="payload-pre-processing"
Sep 17 23:27:15.726842 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.726818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.730413 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.730389    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-fc8f1e\"/\"kube-root-ca.crt\""
Sep 17 23:27:15.730554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.730419    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-fc8f1e\"/\"default-dockercfg-p7bqm\""
Sep 17 23:27:15.730554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.730447    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-fc8f1e\"/\"noten-model-fc8f1e-kserve-self-signed-certs\""
Sep 17 23:27:15.730699 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.730421    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-fc8f1e\"/\"openshift-service-ca.crt\""
Sep 17 23:27:15.736753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.736709    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"]
Sep 17 23:27:15.768853 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.768821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqx7\" (UniqueName: \"kubernetes.io/projected/e8ca2f65-8917-4142-8469-e5f42cf97b17-kube-api-access-tbqx7\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.769091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.768862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-home\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.769091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.768999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-tmp-dir\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.769091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.769039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-dshm\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.769091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.769061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-model-cache\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.769091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.769082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ca2f65-8917-4142-8469-e5f42cf97b17-tls-certs\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870477 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-tmp-dir\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870650 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-dshm\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870650 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-model-cache\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870650 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ca2f65-8917-4142-8469-e5f42cf97b17-tls-certs\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqx7\" (UniqueName: \"kubernetes.io/projected/e8ca2f65-8917-4142-8469-e5f42cf97b17-kube-api-access-tbqx7\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-home\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.870998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.870975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-tmp-dir\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.871068 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.871032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-model-cache\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.871199 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.871168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-home\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.872830 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.872805    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-dshm\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.873119 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.873103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ca2f65-8917-4142-8469-e5f42cf97b17-tls-certs\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:15.879200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:15.879179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqx7\" (UniqueName: \"kubernetes.io/projected/e8ca2f65-8917-4142-8469-e5f42cf97b17-kube-api-access-tbqx7\") pod \"noten-model-fc8f1e-kserve-755ffdb95f-cm5wm\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") " pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:16.040637 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:16.040554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:16.309997 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:16.309973    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"]
Sep 17 23:27:16.312579 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:27:16.312545    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8ca2f65_8917_4142_8469_e5f42cf97b17.slice/crio-e7a47d8ad60f011cdaaca9a305423bacdc04099fef4bdb42a07f1770453fb7e1 WatchSource:0}: Error finding container e7a47d8ad60f011cdaaca9a305423bacdc04099fef4bdb42a07f1770453fb7e1: Status 404 returned error can't find the container with id e7a47d8ad60f011cdaaca9a305423bacdc04099fef4bdb42a07f1770453fb7e1
Sep 17 23:27:16.604610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:16.604510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm" event={"ID":"e8ca2f65-8917-4142-8469-e5f42cf97b17","Type":"ContainerStarted","Data":"65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e"}
Sep 17 23:27:16.604610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:16.604563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm" event={"ID":"e8ca2f65-8917-4142-8469-e5f42cf97b17","Type":"ContainerStarted","Data":"e7a47d8ad60f011cdaaca9a305423bacdc04099fef4bdb42a07f1770453fb7e1"}
Sep 17 23:27:16.621226 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:16.621163    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm" podStartSLOduration=1.621142546 podStartE2EDuration="1.621142546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:16.620235289 +0000 UTC m=+1279.094178274" watchObservedRunningTime="2026-09-17 23:27:16.621142546 +0000 UTC m=+1279.095085529"
Sep 17 23:27:24.846079 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.846047    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"]
Sep 17 23:27:24.849575 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.849558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:24.851982 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.851963    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d69250-dockercfg-4kpt9\""
Sep 17 23:27:24.852054 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.851963    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d69250\""
Sep 17 23:27:24.859035 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.859010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"]
Sep 17 23:27:24.959512 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.959484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8247af-b448-4951-bdb2-76f32b55e666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:24.959623 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:24.959590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vjj\" (UniqueName: \"kubernetes.io/projected/0a8247af-b448-4951-bdb2-76f32b55e666-kube-api-access-26vjj\") pod \"payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.060884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.060858    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vjj\" (UniqueName: \"kubernetes.io/projected/0a8247af-b448-4951-bdb2-76f32b55e666-kube-api-access-26vjj\") pod \"payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.061045 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.060906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8247af-b448-4951-bdb2-76f32b55e666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.061640 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.061620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8247af-b448-4951-bdb2-76f32b55e666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.061799 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.061777    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"]
Sep 17 23:27:25.062022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.061996    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc" podUID="73895d71-0276-4b59-b282-0fd88c6747a3" containerName="payload-processing" containerID="cri-o://b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371" gracePeriod=30
Sep 17 23:27:25.067047 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.067022    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"]
Sep 17 23:27:25.067386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.067360    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752" podUID="cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" containerName="payload-pre-processing" containerID="cri-o://9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025" gracePeriod=30
Sep 17 23:27:25.073708 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.073642    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vjj\" (UniqueName: \"kubernetes.io/projected/0a8247af-b448-4951-bdb2-76f32b55e666-kube-api-access-26vjj\") pod \"payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.160684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.160657    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.363268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.363246    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:27:25.366325 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.366306    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:27:25.439704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.439669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"]
Sep 17 23:27:25.442080 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:27:25.442059    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8247af_b448_4951_bdb2_76f32b55e666.slice/crio-a28ead768e2dc87a630d6f237a10f0cd29d3a7ed571df66cabbe12af4cbc9905 WatchSource:0}: Error finding container a28ead768e2dc87a630d6f237a10f0cd29d3a7ed571df66cabbe12af4cbc9905: Status 404 returned error can't find the container with id a28ead768e2dc87a630d6f237a10f0cd29d3a7ed571df66cabbe12af4cbc9905
Sep 17 23:27:25.464524 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.464503    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p52v\" (UniqueName: \"kubernetes.io/projected/73895d71-0276-4b59-b282-0fd88c6747a3-kube-api-access-2p52v\") pod \"73895d71-0276-4b59-b282-0fd88c6747a3\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") "
Sep 17 23:27:25.464610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.464539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73895d71-0276-4b59-b282-0fd88c6747a3-plugins-config-volume\") pod \"73895d71-0276-4b59-b282-0fd88c6747a3\" (UID: \"73895d71-0276-4b59-b282-0fd88c6747a3\") "
Sep 17 23:27:25.464610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.464568    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzkxl\" (UniqueName: \"kubernetes.io/projected/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-kube-api-access-wzkxl\") pod \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") "
Sep 17 23:27:25.464610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.464587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-plugins-config-volume\") pod \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\" (UID: \"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2\") "
Sep 17 23:27:25.464925 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.464899    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73895d71-0276-4b59-b282-0fd88c6747a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73895d71-0276-4b59-b282-0fd88c6747a3" (UID: "73895d71-0276-4b59-b282-0fd88c6747a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:25.464997 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.464974    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" (UID: "cdbf0b6c-bdd6-462f-a89f-8b52155d25f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:25.466553 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.466533    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-kube-api-access-wzkxl" (OuterVolumeSpecName: "kube-api-access-wzkxl") pod "cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" (UID: "cdbf0b6c-bdd6-462f-a89f-8b52155d25f2"). InnerVolumeSpecName "kube-api-access-wzkxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:25.466616 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.466560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73895d71-0276-4b59-b282-0fd88c6747a3-kube-api-access-2p52v" (OuterVolumeSpecName: "kube-api-access-2p52v") pod "73895d71-0276-4b59-b282-0fd88c6747a3" (UID: "73895d71-0276-4b59-b282-0fd88c6747a3"). InnerVolumeSpecName "kube-api-access-2p52v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:25.565554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.565529    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p52v\" (UniqueName: \"kubernetes.io/projected/73895d71-0276-4b59-b282-0fd88c6747a3-kube-api-access-2p52v\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:25.565554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.565553    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73895d71-0276-4b59-b282-0fd88c6747a3-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:25.565733 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.565562    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzkxl\" (UniqueName: \"kubernetes.io/projected/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-kube-api-access-wzkxl\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:25.565733 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.565570    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:25.656851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.656701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm" event={"ID":"0a8247af-b448-4951-bdb2-76f32b55e666","Type":"ContainerStarted","Data":"8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969"}
Sep 17 23:27:25.656851 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.656782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm" event={"ID":"0a8247af-b448-4951-bdb2-76f32b55e666","Type":"ContainerStarted","Data":"a28ead768e2dc87a630d6f237a10f0cd29d3a7ed571df66cabbe12af4cbc9905"}
Sep 17 23:27:25.657111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.656949    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:25.658201 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.658179    2559 generic.go:356] "Generic (PLEG): container finished" podID="cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" containerID="9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025" exitCode=0
Sep 17 23:27:25.658289 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.658214    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"
Sep 17 23:27:25.658289 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.658265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752" event={"ID":"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2","Type":"ContainerDied","Data":"9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025"}
Sep 17 23:27:25.658394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.658319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752" event={"ID":"cdbf0b6c-bdd6-462f-a89f-8b52155d25f2","Type":"ContainerDied","Data":"b03f551422c86bf11175e190c88255841b17da17a820f7200c8bb7f090b2f936"}
Sep 17 23:27:25.658394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.658351    2559 scope.go:117] "RemoveContainer" containerID="9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025"
Sep 17 23:27:25.659806 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.659781    2559 generic.go:356] "Generic (PLEG): container finished" podID="73895d71-0276-4b59-b282-0fd88c6747a3" containerID="b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371" exitCode=0
Sep 17 23:27:25.659929 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.659825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc" event={"ID":"73895d71-0276-4b59-b282-0fd88c6747a3","Type":"ContainerDied","Data":"b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371"}
Sep 17 23:27:25.660272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.660238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc" event={"ID":"73895d71-0276-4b59-b282-0fd88c6747a3","Type":"ContainerDied","Data":"d08c752421eb4b5d67b63ed856d89efde64bf31888ba7402720d828ffd7a30d6"}
Sep 17 23:27:25.660272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.659832    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"
Sep 17 23:27:25.667371 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.667353    2559 scope.go:117] "RemoveContainer" containerID="9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025"
Sep 17 23:27:25.667621 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:27:25.667599    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025\": container with ID starting with 9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025 not found: ID does not exist" containerID="9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025"
Sep 17 23:27:25.667678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.667630    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025"} err="failed to get container status \"9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025\": rpc error: code = NotFound desc = could not find container \"9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025\": container with ID starting with 9fda8db255c2a225352657c90d443c486aec35776fcbc5497599a17c3e7bb025 not found: ID does not exist"
Sep 17 23:27:25.667678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.667648    2559 scope.go:117] "RemoveContainer" containerID="b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371"
Sep 17 23:27:25.676074 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.676055    2559 scope.go:117] "RemoveContainer" containerID="b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371"
Sep 17 23:27:25.676368 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:27:25.676340    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371\": container with ID starting with b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371 not found: ID does not exist" containerID="b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371"
Sep 17 23:27:25.676468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.676377    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371"} err="failed to get container status \"b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371\": rpc error: code = NotFound desc = could not find container \"b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371\": container with ID starting with b0292b7ef7969fc312a0279b48941da14deff81fa69a3e195bf1635fd0981371 not found: ID does not exist"
Sep 17 23:27:25.686074 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.686037    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm" podStartSLOduration=1.686024564 podStartE2EDuration="1.686024564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:25.673648381 +0000 UTC m=+1288.147591366" watchObservedRunningTime="2026-09-17 23:27:25.686024564 +0000 UTC m=+1288.159967546"
Sep 17 23:27:25.687123 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.687107    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"]
Sep 17 23:27:25.690835 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.690788    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-307a7b-568bb6684c-z9spc"]
Sep 17 23:27:25.700323 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.700302    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"]
Sep 17 23:27:25.703837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:25.703814    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-307a7b-547c75fbfd-24752"]
Sep 17 23:27:26.041575 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:26.041555    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:26.041951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:26.041595    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:26.053967 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:26.053948    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:26.679691 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:26.679660    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:27.641078 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:27.641042    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73895d71-0276-4b59-b282-0fd88c6747a3" path="/var/lib/kubelet/pods/73895d71-0276-4b59-b282-0fd88c6747a3/volumes"
Sep 17 23:27:27.641529 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:27.641513    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" path="/var/lib/kubelet/pods/cdbf0b6c-bdd6-462f-a89f-8b52155d25f2/volumes"
Sep 17 23:27:27.838630 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:27.838599    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:27:28.605633 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:28.605602    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"]
Sep 17 23:27:36.670249 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:36.670212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:27:38.557976 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.557936    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"]
Sep 17 23:27:38.558330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.558285    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" containerName="payload-pre-processing"
Sep 17 23:27:38.558330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.558296    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" containerName="payload-pre-processing"
Sep 17 23:27:38.558330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.558323    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73895d71-0276-4b59-b282-0fd88c6747a3" containerName="payload-processing"
Sep 17 23:27:38.558330 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.558328    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73895d71-0276-4b59-b282-0fd88c6747a3" containerName="payload-processing"
Sep 17 23:27:38.558478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.558388    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbf0b6c-bdd6-462f-a89f-8b52155d25f2" containerName="payload-pre-processing"
Sep 17 23:27:38.558478 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.558398    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="73895d71-0276-4b59-b282-0fd88c6747a3" containerName="payload-processing"
Sep 17 23:27:38.562948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.562929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.569156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.569130    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d90df4\""
Sep 17 23:27:38.569280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.569186    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d90df4-dockercfg-zfxvk\""
Sep 17 23:27:38.578389 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.578368    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"]
Sep 17 23:27:38.592113 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.592091    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"]
Sep 17 23:27:38.601820 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.601800    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.606316 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.606287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"]
Sep 17 23:27:38.672918 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.672893    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.673022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.672929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa04723-12bd-4525-b26f-f6e1d1841ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.673022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.672954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rm9v\" (UniqueName: \"kubernetes.io/projected/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-kube-api-access-6rm9v\") pod \"payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.673022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.673004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xqm\" (UniqueName: \"kubernetes.io/projected/7aa04723-12bd-4525-b26f-f6e1d1841ccb-kube-api-access-r8xqm\") pod \"payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.773848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.773823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa04723-12bd-4525-b26f-f6e1d1841ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.773943 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.773853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rm9v\" (UniqueName: \"kubernetes.io/projected/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-kube-api-access-6rm9v\") pod \"payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.773943 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.773874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xqm\" (UniqueName: \"kubernetes.io/projected/7aa04723-12bd-4525-b26f-f6e1d1841ccb-kube-api-access-r8xqm\") pod \"payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.773943 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.773936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.774578 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.774558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.774630 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.774563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa04723-12bd-4525-b26f-f6e1d1841ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.782005 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.781981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xqm\" (UniqueName: \"kubernetes.io/projected/7aa04723-12bd-4525-b26f-f6e1d1841ccb-kube-api-access-r8xqm\") pod \"payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.782092 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.782073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rm9v\" (UniqueName: \"kubernetes.io/projected/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-kube-api-access-6rm9v\") pod \"payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:38.873122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.873060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:38.912413 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:38.912381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:39.143695 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.143671    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"]
Sep 17 23:27:39.145585 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:27:39.145561    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa04723_12bd_4525_b26f_f6e1d1841ccb.slice/crio-2e3422740524beeb8e12da4f66e291e215062666233c17c33128a5a93ca9737b WatchSource:0}: Error finding container 2e3422740524beeb8e12da4f66e291e215062666233c17c33128a5a93ca9737b: Status 404 returned error can't find the container with id 2e3422740524beeb8e12da4f66e291e215062666233c17c33128a5a93ca9737b
Sep 17 23:27:39.189433 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.189406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"]
Sep 17 23:27:39.191060 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:27:39.191037    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87e9df5_6d4e_40c4_8bc6_f513f29c001a.slice/crio-190b4443739a84118a826cfc84df8259de23a5dff07de34f99b9a84de11bde3a WatchSource:0}: Error finding container 190b4443739a84118a826cfc84df8259de23a5dff07de34f99b9a84de11bde3a: Status 404 returned error can't find the container with id 190b4443739a84118a826cfc84df8259de23a5dff07de34f99b9a84de11bde3a
Sep 17 23:27:39.737935 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.737896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8" event={"ID":"7aa04723-12bd-4525-b26f-f6e1d1841ccb","Type":"ContainerStarted","Data":"a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc"}
Sep 17 23:27:39.738383 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.737956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8" event={"ID":"7aa04723-12bd-4525-b26f-f6e1d1841ccb","Type":"ContainerStarted","Data":"2e3422740524beeb8e12da4f66e291e215062666233c17c33128a5a93ca9737b"}
Sep 17 23:27:39.738383 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.737987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:39.739676 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.739648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp" event={"ID":"b87e9df5-6d4e-40c4-8bc6-f513f29c001a","Type":"ContainerStarted","Data":"fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713"}
Sep 17 23:27:39.739834 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.739699    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:39.739834 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.739740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp" event={"ID":"b87e9df5-6d4e-40c4-8bc6-f513f29c001a","Type":"ContainerStarted","Data":"190b4443739a84118a826cfc84df8259de23a5dff07de34f99b9a84de11bde3a"}
Sep 17 23:27:39.755435 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.755395    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8" podStartSLOduration=1.755380534 podStartE2EDuration="1.755380534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:39.753643503 +0000 UTC m=+1302.227586486" watchObservedRunningTime="2026-09-17 23:27:39.755380534 +0000 UTC m=+1302.229323516"
Sep 17 23:27:39.770767 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:39.770710    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp" podStartSLOduration=1.77069526 podStartE2EDuration="1.77069526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:39.768499852 +0000 UTC m=+1302.242442836" watchObservedRunningTime="2026-09-17 23:27:39.77069526 +0000 UTC m=+1302.244638243"
Sep 17 23:27:43.046106 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.046054    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:27:43.696586 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.696552    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm" podUID="e8ca2f65-8917-4142-8469-e5f42cf97b17" containerName="main" containerID="cri-o://65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e" gracePeriod=45
Sep 17 23:27:43.940111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.940081    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"]
Sep 17 23:27:43.943909 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.943886    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:43.946404 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.946385    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:43.946506 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.946440    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fd82a9\""
Sep 17 23:27:43.946633 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.946583    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fd82a9-dockercfg-r275j\""
Sep 17 23:27:43.955742 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:43.955703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"]
Sep 17 23:27:44.015624 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.015598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.015750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.015627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2w5m\" (UniqueName: \"kubernetes.io/projected/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-kube-api-access-n2w5m\") pod \"payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.116143 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116118    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ca2f65-8917-4142-8469-e5f42cf97b17-tls-certs\") pod \"e8ca2f65-8917-4142-8469-e5f42cf97b17\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") "
Sep 17 23:27:44.116508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116165    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-home\") pod \"e8ca2f65-8917-4142-8469-e5f42cf97b17\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") "
Sep 17 23:27:44.116508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116228    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbqx7\" (UniqueName: \"kubernetes.io/projected/e8ca2f65-8917-4142-8469-e5f42cf97b17-kube-api-access-tbqx7\") pod \"e8ca2f65-8917-4142-8469-e5f42cf97b17\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") "
Sep 17 23:27:44.116508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116307    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-dshm\") pod \"e8ca2f65-8917-4142-8469-e5f42cf97b17\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") "
Sep 17 23:27:44.116508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-model-cache\") pod \"e8ca2f65-8917-4142-8469-e5f42cf97b17\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") "
Sep 17 23:27:44.116508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-tmp-dir\") pod \"e8ca2f65-8917-4142-8469-e5f42cf97b17\" (UID: \"e8ca2f65-8917-4142-8469-e5f42cf97b17\") "
Sep 17 23:27:44.116508 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116425    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-home" (OuterVolumeSpecName: "home") pod "e8ca2f65-8917-4142-8469-e5f42cf97b17" (UID: "e8ca2f65-8917-4142-8469-e5f42cf97b17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116630    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8ca2f65-8917-4142-8469-e5f42cf97b17" (UID: "e8ca2f65-8917-4142-8469-e5f42cf97b17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116646    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8ca2f65-8917-4142-8469-e5f42cf97b17" (UID: "e8ca2f65-8917-4142-8469-e5f42cf97b17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2w5m\" (UniqueName: \"kubernetes.io/projected/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-kube-api-access-n2w5m\") pod \"payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116757    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116774    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:44.116836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.116787    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:44.117326 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.117307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.118579 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.118557    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-dshm" (OuterVolumeSpecName: "dshm") pod "e8ca2f65-8917-4142-8469-e5f42cf97b17" (UID: "e8ca2f65-8917-4142-8469-e5f42cf97b17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:44.118943 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.118919    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ca2f65-8917-4142-8469-e5f42cf97b17-kube-api-access-tbqx7" (OuterVolumeSpecName: "kube-api-access-tbqx7") pod "e8ca2f65-8917-4142-8469-e5f42cf97b17" (UID: "e8ca2f65-8917-4142-8469-e5f42cf97b17"). InnerVolumeSpecName "kube-api-access-tbqx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:44.118943 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.118921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ca2f65-8917-4142-8469-e5f42cf97b17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8ca2f65-8917-4142-8469-e5f42cf97b17" (UID: "e8ca2f65-8917-4142-8469-e5f42cf97b17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:44.123556 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.123536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2w5m\" (UniqueName: \"kubernetes.io/projected/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-kube-api-access-n2w5m\") pod \"payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.217353 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.217329    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqx7\" (UniqueName: \"kubernetes.io/projected/e8ca2f65-8917-4142-8469-e5f42cf97b17-kube-api-access-tbqx7\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:44.217353 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.217351    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ca2f65-8917-4142-8469-e5f42cf97b17-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:44.217510 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.217361    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ca2f65-8917-4142-8469-e5f42cf97b17-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:27:44.253891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.253867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.528744 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.528701    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"]
Sep 17 23:27:44.530929 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:27:44.530905    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66f453d_1834_4461_84d9_4ca6ed9b2ae3.slice/crio-465106e740332fe419b3d02a7ecd419f4d852b91f7fbbeadf0a1c04a19a8d5bb WatchSource:0}: Error finding container 465106e740332fe419b3d02a7ecd419f4d852b91f7fbbeadf0a1c04a19a8d5bb: Status 404 returned error can't find the container with id 465106e740332fe419b3d02a7ecd419f4d852b91f7fbbeadf0a1c04a19a8d5bb
Sep 17 23:27:44.767861 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.767829    2559 generic.go:356] "Generic (PLEG): container finished" podID="e8ca2f65-8917-4142-8469-e5f42cf97b17" containerID="65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e" exitCode=0
Sep 17 23:27:44.768094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.767885    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"
Sep 17 23:27:44.768094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.767919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm" event={"ID":"e8ca2f65-8917-4142-8469-e5f42cf97b17","Type":"ContainerDied","Data":"65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e"}
Sep 17 23:27:44.768094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.767977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm" event={"ID":"e8ca2f65-8917-4142-8469-e5f42cf97b17","Type":"ContainerDied","Data":"e7a47d8ad60f011cdaaca9a305423bacdc04099fef4bdb42a07f1770453fb7e1"}
Sep 17 23:27:44.768094 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.768005    2559 scope.go:117] "RemoveContainer" containerID="65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e"
Sep 17 23:27:44.770017 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.769997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94" event={"ID":"b66f453d-1834-4461-84d9-4ca6ed9b2ae3","Type":"ContainerStarted","Data":"783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a"}
Sep 17 23:27:44.770144 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.770038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94" event={"ID":"b66f453d-1834-4461-84d9-4ca6ed9b2ae3","Type":"ContainerStarted","Data":"465106e740332fe419b3d02a7ecd419f4d852b91f7fbbeadf0a1c04a19a8d5bb"}
Sep 17 23:27:44.770144 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.770107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:44.777111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.777094    2559 scope.go:117] "RemoveContainer" containerID="65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e"
Sep 17 23:27:44.777418 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:27:44.777394    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e\": container with ID starting with 65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e not found: ID does not exist" containerID="65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e"
Sep 17 23:27:44.777512 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.777427    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e"} err="failed to get container status \"65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e\": rpc error: code = NotFound desc = could not find container \"65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e\": container with ID starting with 65834e79c34512604fc671a055f3414c4f73623c5b1a0a59ed864bb36cc8728e not found: ID does not exist"
Sep 17 23:27:44.786651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.786538    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94" podStartSLOduration=1.786514763 podStartE2EDuration="1.786514763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:44.786449052 +0000 UTC m=+1307.260392037" watchObservedRunningTime="2026-09-17 23:27:44.786514763 +0000 UTC m=+1307.260457747"
Sep 17 23:27:44.799410 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.799382    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"]
Sep 17 23:27:44.804294 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:44.804274    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-fc8f1e/noten-model-fc8f1e-kserve-755ffdb95f-cm5wm"]
Sep 17 23:27:45.640026 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:45.639995    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ca2f65-8917-4142-8469-e5f42cf97b17" path="/var/lib/kubelet/pods/e8ca2f65-8917-4142-8469-e5f42cf97b17/volumes"
Sep 17 23:27:50.746929 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:50.746879    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:27:50.747450 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:50.747056    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:27:55.778762 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:55.778707    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:27:56.840480 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:56.840434    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:27:59.718757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:27:59.718704    2559 scope.go:117] "RemoveContainer" containerID="68d58a2e74c0eb50aba638607b166e4084a4f1142913b15614cd153b8d6082c5"
Sep 17 23:28:06.528116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.528085    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7849-f5e795"]
Sep 17 23:28:06.528497 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.528483    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ca2f65-8917-4142-8469-e5f42cf97b17" containerName="main"
Sep 17 23:28:06.528541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.528498    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ca2f65-8917-4142-8469-e5f42cf97b17" containerName="main"
Sep 17 23:28:06.528583 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.528553    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ca2f65-8917-4142-8469-e5f42cf97b17" containerName="main"
Sep 17 23:28:06.533411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.533384    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:06.535735 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.535697    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6p2p\""
Sep 17 23:28:06.536167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.536133    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-f5e795"]
Sep 17 23:28:06.601805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.601781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfdlv\" (UniqueName: \"kubernetes.io/projected/50e072c7-ecc6-4a66-bec3-cf825e1343db-kube-api-access-wfdlv\") pod \"test-curl-7849-f5e795\" (UID: \"50e072c7-ecc6-4a66-bec3-cf825e1343db\") " pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:06.702267 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.702245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfdlv\" (UniqueName: \"kubernetes.io/projected/50e072c7-ecc6-4a66-bec3-cf825e1343db-kube-api-access-wfdlv\") pod \"test-curl-7849-f5e795\" (UID: \"50e072c7-ecc6-4a66-bec3-cf825e1343db\") " pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:06.711998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.711971    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfdlv\" (UniqueName: \"kubernetes.io/projected/50e072c7-ecc6-4a66-bec3-cf825e1343db-kube-api-access-wfdlv\") pod \"test-curl-7849-f5e795\" (UID: \"50e072c7-ecc6-4a66-bec3-cf825e1343db\") " pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:06.842956 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.842892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:06.896662 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.896250    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"]
Sep 17 23:28:06.901146 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.901123    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:06.908300 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.907044    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d92920-dockercfg-56vwx\""
Sep 17 23:28:06.908300 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.907323    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d92920\""
Sep 17 23:28:06.912572 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.912243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"]
Sep 17 23:28:06.935069 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.935046    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"]
Sep 17 23:28:06.938506 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.938485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:06.948563 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:06.948538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"]
Sep 17 23:28:07.005255 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.005229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7d3d6c-7876-4c62-91d4-d81c83d40969-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.005371 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.005281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72sr\" (UniqueName: \"kubernetes.io/projected/5c7d3d6c-7876-4c62-91d4-d81c83d40969-kube-api-access-b72sr\") pod \"payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.005371 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.005345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc91222-888f-47a5-aa12-af6164fe2738-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.005453 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.005414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtk4n\" (UniqueName: \"kubernetes.io/projected/ccc91222-888f-47a5-aa12-af6164fe2738-kube-api-access-jtk4n\") pod \"payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.106400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.106345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc91222-888f-47a5-aa12-af6164fe2738-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.106400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.106389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtk4n\" (UniqueName: \"kubernetes.io/projected/ccc91222-888f-47a5-aa12-af6164fe2738-kube-api-access-jtk4n\") pod \"payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.106543 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.106415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7d3d6c-7876-4c62-91d4-d81c83d40969-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.106543 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.106475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72sr\" (UniqueName: \"kubernetes.io/projected/5c7d3d6c-7876-4c62-91d4-d81c83d40969-kube-api-access-b72sr\") pod \"payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.107099 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.107082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc91222-888f-47a5-aa12-af6164fe2738-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.107150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.107082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7d3d6c-7876-4c62-91d4-d81c83d40969-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.110707 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.110688    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-f5e795"]
Sep 17 23:28:07.113122 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:07.113097    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50e072c7_ecc6_4a66_bec3_cf825e1343db.slice/crio-4370e9a43068c75e26f213feffedf74d85af22a83912f3bb2ef0f67129011041 WatchSource:0}: Error finding container 4370e9a43068c75e26f213feffedf74d85af22a83912f3bb2ef0f67129011041: Status 404 returned error can't find the container with id 4370e9a43068c75e26f213feffedf74d85af22a83912f3bb2ef0f67129011041
Sep 17 23:28:07.114268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.114245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72sr\" (UniqueName: \"kubernetes.io/projected/5c7d3d6c-7876-4c62-91d4-d81c83d40969-kube-api-access-b72sr\") pod \"payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.114388 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.114369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtk4n\" (UniqueName: \"kubernetes.io/projected/ccc91222-888f-47a5-aa12-af6164fe2738-kube-api-access-jtk4n\") pod \"payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.216100 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.216071    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.249380 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.249352    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.487283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.487260    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"]
Sep 17 23:28:07.489142 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:07.489114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc91222_888f_47a5_aa12_af6164fe2738.slice/crio-4d31f2f0e98aa621729a6af7424d064a05c9c7522a41c456dd36052a0e940531 WatchSource:0}: Error finding container 4d31f2f0e98aa621729a6af7424d064a05c9c7522a41c456dd36052a0e940531: Status 404 returned error can't find the container with id 4d31f2f0e98aa621729a6af7424d064a05c9c7522a41c456dd36052a0e940531
Sep 17 23:28:07.535948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.535911    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"]
Sep 17 23:28:07.538173 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:07.538141    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7d3d6c_7876_4c62_91d4_d81c83d40969.slice/crio-30f92044b1148d801e5aa0a207f37869eba4499cca501ff613a5d09d4d7e03cf WatchSource:0}: Error finding container 30f92044b1148d801e5aa0a207f37869eba4499cca501ff613a5d09d4d7e03cf: Status 404 returned error can't find the container with id 30f92044b1148d801e5aa0a207f37869eba4499cca501ff613a5d09d4d7e03cf
Sep 17 23:28:07.903440 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.903366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s" event={"ID":"5c7d3d6c-7876-4c62-91d4-d81c83d40969","Type":"ContainerStarted","Data":"8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729"}
Sep 17 23:28:07.903440 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.903425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s" event={"ID":"5c7d3d6c-7876-4c62-91d4-d81c83d40969","Type":"ContainerStarted","Data":"30f92044b1148d801e5aa0a207f37869eba4499cca501ff613a5d09d4d7e03cf"}
Sep 17 23:28:07.903654 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.903456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:07.905056 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.905033    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-f5e795" event={"ID":"50e072c7-ecc6-4a66-bec3-cf825e1343db","Type":"ContainerStarted","Data":"035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738"}
Sep 17 23:28:07.905190 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.905067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-f5e795" event={"ID":"50e072c7-ecc6-4a66-bec3-cf825e1343db","Type":"ContainerStarted","Data":"4370e9a43068c75e26f213feffedf74d85af22a83912f3bb2ef0f67129011041"}
Sep 17 23:28:07.906688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.906665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg" event={"ID":"ccc91222-888f-47a5-aa12-af6164fe2738","Type":"ContainerStarted","Data":"63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5"}
Sep 17 23:28:07.906821 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.906701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg" event={"ID":"ccc91222-888f-47a5-aa12-af6164fe2738","Type":"ContainerStarted","Data":"4d31f2f0e98aa621729a6af7424d064a05c9c7522a41c456dd36052a0e940531"}
Sep 17 23:28:07.906821 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.906733    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:07.919816 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.919777    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s" podStartSLOduration=1.919764374 podStartE2EDuration="1.919764374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:07.918439286 +0000 UTC m=+1330.392382268" watchObservedRunningTime="2026-09-17 23:28:07.919764374 +0000 UTC m=+1330.393707351"
Sep 17 23:28:07.933748 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.933693    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg" podStartSLOduration=1.933679983 podStartE2EDuration="1.933679983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:07.932171928 +0000 UTC m=+1330.406114903" watchObservedRunningTime="2026-09-17 23:28:07.933679983 +0000 UTC m=+1330.407622966"
Sep 17 23:28:07.947936 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:07.947901    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7849-f5e795" podStartSLOduration=1.947889565 podStartE2EDuration="1.947889565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:07.945936317 +0000 UTC m=+1330.419879302" watchObservedRunningTime="2026-09-17 23:28:07.947889565 +0000 UTC m=+1330.421832593"
Sep 17 23:28:08.577189 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.577163    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7849-f5e795"]
Sep 17 23:28:08.592653 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.592622    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7849-f5e795"]
Sep 17 23:28:08.711326 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.711295    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7849-f31621"]
Sep 17 23:28:08.711671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.711656    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e072c7-ecc6-4a66-bec3-cf825e1343db" containerName="test-curl-7849-f5e795"
Sep 17 23:28:08.711671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.711671    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e072c7-ecc6-4a66-bec3-cf825e1343db" containerName="test-curl-7849-f5e795"
Sep 17 23:28:08.711811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.711762    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e072c7-ecc6-4a66-bec3-cf825e1343db" containerName="test-curl-7849-f5e795"
Sep 17 23:28:08.714868 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.714853    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:08.720894 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.720863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-f31621"]
Sep 17 23:28:08.821963 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.821938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhb4\" (UniqueName: \"kubernetes.io/projected/2176ae19-acfb-4732-9d25-74eba4bf6014-kube-api-access-dqhb4\") pod \"test-curl-7849-f31621\" (UID: \"2176ae19-acfb-4732-9d25-74eba4bf6014\") " pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:08.911693 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:28:08.911667    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7849-f5e795" container="test-curl-7849-f5e795"
Sep 17 23:28:08.923272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.923250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhb4\" (UniqueName: \"kubernetes.io/projected/2176ae19-acfb-4732-9d25-74eba4bf6014-kube-api-access-dqhb4\") pod \"test-curl-7849-f31621\" (UID: \"2176ae19-acfb-4732-9d25-74eba4bf6014\") " pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:08.931098 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:08.931071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhb4\" (UniqueName: \"kubernetes.io/projected/2176ae19-acfb-4732-9d25-74eba4bf6014-kube-api-access-dqhb4\") pod \"test-curl-7849-f31621\" (UID: \"2176ae19-acfb-4732-9d25-74eba4bf6014\") " pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:09.027660 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:09.027605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:09.295520 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:09.295490    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-f31621"]
Sep 17 23:28:09.297811 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:09.297775    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2176ae19_acfb_4732_9d25_74eba4bf6014.slice/crio-7ce982237d5321bc870da68e6f794c97fe754f92ddfe9eab7e0eb4068b907284 WatchSource:0}: Error finding container 7ce982237d5321bc870da68e6f794c97fe754f92ddfe9eab7e0eb4068b907284: Status 404 returned error can't find the container with id 7ce982237d5321bc870da68e6f794c97fe754f92ddfe9eab7e0eb4068b907284
Sep 17 23:28:09.918360 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:09.918319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-f31621" event={"ID":"2176ae19-acfb-4732-9d25-74eba4bf6014","Type":"ContainerStarted","Data":"022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed"}
Sep 17 23:28:09.918800 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:09.918375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-f31621" event={"ID":"2176ae19-acfb-4732-9d25-74eba4bf6014","Type":"ContainerStarted","Data":"7ce982237d5321bc870da68e6f794c97fe754f92ddfe9eab7e0eb4068b907284"}
Sep 17 23:28:09.918800 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:09.918446    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7849-f5e795" podUID="50e072c7-ecc6-4a66-bec3-cf825e1343db" containerName="test-curl-7849-f5e795" containerID="cri-o://035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738" gracePeriod=2
Sep 17 23:28:09.932409 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:09.932368    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7849-f31621" podStartSLOduration=1.9323484739999999 podStartE2EDuration="1.932348474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:09.930499491 +0000 UTC m=+1332.404442479" watchObservedRunningTime="2026-09-17 23:28:09.932348474 +0000 UTC m=+1332.406291459"
Sep 17 23:28:10.518305 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:10.518271    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7849-f31621"]
Sep 17 23:28:10.520891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:10.520863    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7849-f31621"]
Sep 17 23:28:10.925373 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:28:10.925345    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7849-f31621" container="test-curl-7849-f31621"
Sep 17 23:28:11.930810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:11.930773    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7849-f31621" podUID="2176ae19-acfb-4732-9d25-74eba4bf6014" containerName="test-curl-7849-f31621" containerID="cri-o://022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed" gracePeriod=2
Sep 17 23:28:12.032537 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.032510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:28:12.103776 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.103753    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:12.147416 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.147389    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfdlv\" (UniqueName: \"kubernetes.io/projected/50e072c7-ecc6-4a66-bec3-cf825e1343db-kube-api-access-wfdlv\") pod \"50e072c7-ecc6-4a66-bec3-cf825e1343db\" (UID: \"50e072c7-ecc6-4a66-bec3-cf825e1343db\") "
Sep 17 23:28:12.149772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.149746    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e072c7-ecc6-4a66-bec3-cf825e1343db-kube-api-access-wfdlv" (OuterVolumeSpecName: "kube-api-access-wfdlv") pod "50e072c7-ecc6-4a66-bec3-cf825e1343db" (UID: "50e072c7-ecc6-4a66-bec3-cf825e1343db"). InnerVolumeSpecName "kube-api-access-wfdlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:12.248552 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.248531    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfdlv\" (UniqueName: \"kubernetes.io/projected/50e072c7-ecc6-4a66-bec3-cf825e1343db-kube-api-access-wfdlv\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:28:12.937357 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.937327    2559 generic.go:356] "Generic (PLEG): container finished" podID="50e072c7-ecc6-4a66-bec3-cf825e1343db" containerID="035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738" exitCode=137
Sep 17 23:28:12.937706 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.937373    2559 scope.go:117] "RemoveContainer" containerID="035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738"
Sep 17 23:28:12.937706 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.937372    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:12.946837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.946818    2559 scope.go:117] "RemoveContainer" containerID="035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738"
Sep 17 23:28:12.947115 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:28:12.947096    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738\": container with ID starting with 035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738 not found: ID does not exist" containerID="035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738"
Sep 17 23:28:12.947161 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:12.947126    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738"} err="failed to get container status \"035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738\": rpc error: code = NotFound desc = could not find container \"035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738\": container with ID starting with 035139e2b7a584e1a45b340fc56f943352a6c4247b6150e6a7a7bb3a8dec8738 not found: ID does not exist"
Sep 17 23:28:12.948323 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:28:12.948301    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7849-f5e795\" is forbidden: User \"system:node:ip-10-0-143-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-250.ec2.internal' and this object" podUID="50e072c7-ecc6-4a66-bec3-cf825e1343db" pod="opendatahub/test-curl-7849-f5e795"
Sep 17 23:28:13.639355 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:13.639327    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e072c7-ecc6-4a66-bec3-cf825e1343db" path="/var/lib/kubelet/pods/50e072c7-ecc6-4a66-bec3-cf825e1343db/volumes"
Sep 17 23:28:14.102736 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.102694    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:14.161909 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.161886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqhb4\" (UniqueName: \"kubernetes.io/projected/2176ae19-acfb-4732-9d25-74eba4bf6014-kube-api-access-dqhb4\") pod \"2176ae19-acfb-4732-9d25-74eba4bf6014\" (UID: \"2176ae19-acfb-4732-9d25-74eba4bf6014\") "
Sep 17 23:28:14.163826 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.163803    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2176ae19-acfb-4732-9d25-74eba4bf6014-kube-api-access-dqhb4" (OuterVolumeSpecName: "kube-api-access-dqhb4") pod "2176ae19-acfb-4732-9d25-74eba4bf6014" (UID: "2176ae19-acfb-4732-9d25-74eba4bf6014"). InnerVolumeSpecName "kube-api-access-dqhb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:14.263244 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.263218    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhb4\" (UniqueName: \"kubernetes.io/projected/2176ae19-acfb-4732-9d25-74eba4bf6014-kube-api-access-dqhb4\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:28:14.949825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.949791    2559 generic.go:356] "Generic (PLEG): container finished" podID="2176ae19-acfb-4732-9d25-74eba4bf6014" containerID="022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed" exitCode=137
Sep 17 23:28:14.949985 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.949828    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:14.949985 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.949891    2559 scope.go:117] "RemoveContainer" containerID="022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed"
Sep 17 23:28:14.958772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.958753    2559 scope.go:117] "RemoveContainer" containerID="022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed"
Sep 17 23:28:14.959074 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:28:14.959055    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed\": container with ID starting with 022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed not found: ID does not exist" containerID="022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed"
Sep 17 23:28:14.959123 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:14.959084    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed"} err="failed to get container status \"022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed\": rpc error: code = NotFound desc = could not find container \"022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed\": container with ID starting with 022a85869f02efc438fc3bbca5dfe4177495a3a198f938a7ddd148292a3439ed not found: ID does not exist"
Sep 17 23:28:14.959894 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:28:14.959873    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7849-f31621\" is forbidden: User \"system:node:ip-10-0-143-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-250.ec2.internal' and this object" podUID="2176ae19-acfb-4732-9d25-74eba4bf6014" pod="opendatahub/test-curl-7849-f31621"
Sep 17 23:28:15.640953 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:15.640912    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2176ae19-acfb-4732-9d25-74eba4bf6014" path="/var/lib/kubelet/pods/2176ae19-acfb-4732-9d25-74eba4bf6014/volumes"
Sep 17 23:28:18.914032 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:18.914002    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:28:18.914428 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:18.914084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:28:22.603008 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.602972    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"]
Sep 17 23:28:22.603521 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.603497    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2176ae19-acfb-4732-9d25-74eba4bf6014" containerName="test-curl-7849-f31621"
Sep 17 23:28:22.603665 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.603585    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2176ae19-acfb-4732-9d25-74eba4bf6014" containerName="test-curl-7849-f31621"
Sep 17 23:28:22.603754 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.603741    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2176ae19-acfb-4732-9d25-74eba4bf6014" containerName="test-curl-7849-f31621"
Sep 17 23:28:22.606936 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.606917    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.609070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.609047    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1b9e74c5-dockercfg-2hzc8\""
Sep 17 23:28:22.609070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.609061    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1b9e74c5\""
Sep 17 23:28:22.615700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.615676    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"]
Sep 17 23:28:22.619528 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.619502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"]
Sep 17 23:28:22.619632 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.619611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.629276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.629255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"]
Sep 17 23:28:22.737431 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.737401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5ll\" (UniqueName: \"kubernetes.io/projected/8bd5ce5d-b560-4f96-a699-7e839c693d09-kube-api-access-8f5ll\") pod \"payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.737557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.737444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707d1d6a-4be5-4765-b8c8-59cfe512fd05-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") " pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.737557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.737521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8v4m\" (UniqueName: \"kubernetes.io/projected/707d1d6a-4be5-4765-b8c8-59cfe512fd05-kube-api-access-v8v4m\") pod \"payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") " pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.737557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.737550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5ce5d-b560-4f96-a699-7e839c693d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.838702 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.838674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8v4m\" (UniqueName: \"kubernetes.io/projected/707d1d6a-4be5-4765-b8c8-59cfe512fd05-kube-api-access-v8v4m\") pod \"payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") " pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.838831 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.838705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5ce5d-b560-4f96-a699-7e839c693d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.838831 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.838806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5ll\" (UniqueName: \"kubernetes.io/projected/8bd5ce5d-b560-4f96-a699-7e839c693d09-kube-api-access-8f5ll\") pod \"payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.838960 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.838836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707d1d6a-4be5-4765-b8c8-59cfe512fd05-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") " pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.839709 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.839686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5ce5d-b560-4f96-a699-7e839c693d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.839792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.839686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707d1d6a-4be5-4765-b8c8-59cfe512fd05-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") " pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.849341 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.849307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5ll\" (UniqueName: \"kubernetes.io/projected/8bd5ce5d-b560-4f96-a699-7e839c693d09-kube-api-access-8f5ll\") pod \"payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.849432 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.849378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8v4m\" (UniqueName: \"kubernetes.io/projected/707d1d6a-4be5-4765-b8c8-59cfe512fd05-kube-api-access-v8v4m\") pod \"payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") " pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:22.918194 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.918138    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:22.930129 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:22.930109    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:23.190627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:23.190594    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"]
Sep 17 23:28:23.193398 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:23.193373    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd5ce5d_b560_4f96_a699_7e839c693d09.slice/crio-d6ad6ff5f64d7baba118d49b181e14003d855a55964bd71ac5ce5001f22310a0 WatchSource:0}: Error finding container d6ad6ff5f64d7baba118d49b181e14003d855a55964bd71ac5ce5001f22310a0: Status 404 returned error can't find the container with id d6ad6ff5f64d7baba118d49b181e14003d855a55964bd71ac5ce5001f22310a0
Sep 17 23:28:23.210572 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:23.210486    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"]
Sep 17 23:28:23.211912 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:23.211887    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707d1d6a_4be5_4765_b8c8_59cfe512fd05.slice/crio-da7c0e283a43185d075e2b44cb68231724d6cc9a056c1342f933dd33c3b76840 WatchSource:0}: Error finding container da7c0e283a43185d075e2b44cb68231724d6cc9a056c1342f933dd33c3b76840: Status 404 returned error can't find the container with id da7c0e283a43185d075e2b44cb68231724d6cc9a056c1342f933dd33c3b76840
Sep 17 23:28:24.021674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.021637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m" event={"ID":"707d1d6a-4be5-4765-b8c8-59cfe512fd05","Type":"ContainerStarted","Data":"cf7dc56d6a6c1db635891b081153bb02648a486b4f59c05bce5e78c4a918a314"}
Sep 17 23:28:24.022139 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.021756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m" event={"ID":"707d1d6a-4be5-4765-b8c8-59cfe512fd05","Type":"ContainerStarted","Data":"da7c0e283a43185d075e2b44cb68231724d6cc9a056c1342f933dd33c3b76840"}
Sep 17 23:28:24.022139 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.021787    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:24.023411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.023381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh" event={"ID":"8bd5ce5d-b560-4f96-a699-7e839c693d09","Type":"ContainerStarted","Data":"b72151e83f22c0487a841ef2701ae1a0967fe7194b5b229f259dabc18e57e6cf"}
Sep 17 23:28:24.023543 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.023429    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:24.023543 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.023449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh" event={"ID":"8bd5ce5d-b560-4f96-a699-7e839c693d09","Type":"ContainerStarted","Data":"d6ad6ff5f64d7baba118d49b181e14003d855a55964bd71ac5ce5001f22310a0"}
Sep 17 23:28:24.038696 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.038648    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m" podStartSLOduration=2.038634671 podStartE2EDuration="2.038634671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:24.037804777 +0000 UTC m=+1346.511747753" watchObservedRunningTime="2026-09-17 23:28:24.038634671 +0000 UTC m=+1346.512577668"
Sep 17 23:28:24.053938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:24.053899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh" podStartSLOduration=2.053884802 podStartE2EDuration="2.053884802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:24.051926721 +0000 UTC m=+1346.525869705" watchObservedRunningTime="2026-09-17 23:28:24.053884802 +0000 UTC m=+1346.527827785"
Sep 17 23:28:26.245991 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:26.245953    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:28:35.030371 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:35.030343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:28:35.030818 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:35.030405    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:28:43.734150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.734116    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"]
Sep 17 23:28:43.739678 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.739660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.749707 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.749519    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"kube-root-ca.crt\""
Sep 17 23:28:43.749707 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.749574    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"default-dockercfg-6zl8d\""
Sep 17 23:28:43.749917 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.749856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"auth-test-model-d90df4-kserve-self-signed-certs\""
Sep 17 23:28:43.750129 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.750111    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"openshift-service-ca.crt\""
Sep 17 23:28:43.752383 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.752358    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"]
Sep 17 23:28:43.803914 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.803889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4cs\" (UniqueName: \"kubernetes.io/projected/d9ee0094-1e9d-4677-8c27-d153ea5af8af-kube-api-access-lh4cs\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.804014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.803917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-home\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.804014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.803959    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-model-cache\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.804014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.803984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tls-certs\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.804014 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.804003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tmp-dir\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.804153 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.804107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-dshm\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.904981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.904958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-model-cache\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905079 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.904994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tls-certs\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905079 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tmp-dir\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905079 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-dshm\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905189 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4cs\" (UniqueName: \"kubernetes.io/projected/d9ee0094-1e9d-4677-8c27-d153ea5af8af-kube-api-access-lh4cs\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905189 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-home\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905356 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-model-cache\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905356 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tmp-dir\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.905544 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.905461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-home\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.907223 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.907201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-dshm\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.907591 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.907575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tls-certs\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:43.912328 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:43.912300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4cs\" (UniqueName: \"kubernetes.io/projected/d9ee0094-1e9d-4677-8c27-d153ea5af8af-kube-api-access-lh4cs\") pod \"auth-test-model-d90df4-kserve-585b45db55-267l9\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:44.058266 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:44.058211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:44.330167 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:44.330143    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"]
Sep 17 23:28:44.331881 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:28:44.331855    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ee0094_1e9d_4677_8c27_d153ea5af8af.slice/crio-5bd02abf30ba3174770fc833ca930f92917f4f8c424267becd9d87f0d28fe74c WatchSource:0}: Error finding container 5bd02abf30ba3174770fc833ca930f92917f4f8c424267becd9d87f0d28fe74c: Status 404 returned error can't find the container with id 5bd02abf30ba3174770fc833ca930f92917f4f8c424267becd9d87f0d28fe74c
Sep 17 23:28:45.133334 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:45.133297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9" event={"ID":"d9ee0094-1e9d-4677-8c27-d153ea5af8af","Type":"ContainerStarted","Data":"c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8"}
Sep 17 23:28:45.133781 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:45.133343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9" event={"ID":"d9ee0094-1e9d-4677-8c27-d153ea5af8af","Type":"ContainerStarted","Data":"5bd02abf30ba3174770fc833ca930f92917f4f8c424267becd9d87f0d28fe74c"}
Sep 17 23:28:45.151242 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:45.151190    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9" podStartSLOduration=2.151170827 podStartE2EDuration="2.151170827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:45.150014567 +0000 UTC m=+1367.623957548" watchObservedRunningTime="2026-09-17 23:28:45.151170827 +0000 UTC m=+1367.625113812"
Sep 17 23:28:48.020645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:48.020613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:28:54.058528 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:54.058475    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:54.058937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:54.058548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:54.070332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:54.070311    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:28:54.187651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:28:54.187624    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:29:12.420620 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:12.420586    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:29:49.330671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.330589    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"]
Sep 17 23:29:49.334594 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.334572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.337629 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.337588    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"openshift-service-ca.crt\""
Sep 17 23:29:49.337629 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.337587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"auth-test-model-d92920-kserve-self-signed-certs\""
Sep 17 23:29:49.337916 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.337591    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"default-dockercfg-mwr9r\""
Sep 17 23:29:49.337916 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.337597    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"kube-root-ca.crt\""
Sep 17 23:29:49.341283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.341255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"]
Sep 17 23:29:49.347466 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.347444    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:29:49.438808 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.438787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-home\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.438951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.438833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvj9\" (UniqueName: \"kubernetes.io/projected/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-kube-api-access-xvvj9\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.438951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.438867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-model-cache\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.438951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.438890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-dshm\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.438951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.438912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tmp-dir\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.438951 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.438938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tls-certs\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.539596 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.539566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvj9\" (UniqueName: \"kubernetes.io/projected/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-kube-api-access-xvvj9\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.539764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.539601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-model-cache\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.539764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.539622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-dshm\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.539764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.539643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tmp-dir\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.539764 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.539747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tls-certs\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.540007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.539868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-home\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.540067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.540016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tmp-dir\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.540067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.540044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-model-cache\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.540177 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.540162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-home\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.542011 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.541989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-dshm\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.542260 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.542245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tls-certs\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.546624 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.546602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvj9\" (UniqueName: \"kubernetes.io/projected/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-kube-api-access-xvvj9\") pod \"auth-test-model-d92920-kserve-69b84ccc47-2wxdn\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") " pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.646131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.646071    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:49.916884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:49.916856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"]
Sep 17 23:29:49.919352 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:29:49.919324    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fd8d4c1_5c23_4b23_b681_a53f487d0b9c.slice/crio-33f015b96cf204d3012665be34925a8b9ebc26ede6387c51a3314af052c03722 WatchSource:0}: Error finding container 33f015b96cf204d3012665be34925a8b9ebc26ede6387c51a3314af052c03722: Status 404 returned error can't find the container with id 33f015b96cf204d3012665be34925a8b9ebc26ede6387c51a3314af052c03722
Sep 17 23:29:50.467472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:50.467433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn" event={"ID":"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c","Type":"ContainerStarted","Data":"61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a"}
Sep 17 23:29:50.467472 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:50.467478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn" event={"ID":"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c","Type":"ContainerStarted","Data":"33f015b96cf204d3012665be34925a8b9ebc26ede6387c51a3314af052c03722"}
Sep 17 23:29:50.487124 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:50.487075    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn" podStartSLOduration=1.4870609639999999 podStartE2EDuration="1.487060964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:50.484885895 +0000 UTC m=+1432.958828877" watchObservedRunningTime="2026-09-17 23:29:50.487060964 +0000 UTC m=+1432.961003947"
Sep 17 23:29:59.647022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:59.646991    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:59.647385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:59.647041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:59.659635 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:59.659607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:29:59.776141 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:29:59.776111    2559 scope.go:117] "RemoveContainer" containerID="9722f78b0623b3450aba3feb1d1dd6fde05fffff909169c220f5628ac8b37762"
Sep 17 23:30:00.139053 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.139021    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"]
Sep 17 23:30:00.144064 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.144040    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"]
Sep 17 23:30:00.144220 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.144176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"
Sep 17 23:30:00.146444 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.146421    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\""
Sep 17 23:30:00.147585 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.147565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"
Sep 17 23:30:00.153276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.153205    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"]
Sep 17 23:30:00.155379 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.155358    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"]
Sep 17 23:30:00.159301 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.159278    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"
Sep 17 23:30:00.160828 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.160808    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"]
Sep 17 23:30:00.164903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.164878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"
Sep 17 23:30:00.167168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.167145    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"
Sep 17 23:30:00.171945 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.171920    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"]
Sep 17 23:30:00.209749 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.209681    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"
Sep 17 23:30:00.459193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.459159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"]
Sep 17 23:30:00.460898 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:30:00.460868    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26631a94_c281_47c2_a9cd_47cfed5c0f71.slice/crio-9ba97e68bd6f6a4b8ad27817f93950c1321d8efba45ef3535a17cf020d60cbb0 WatchSource:0}: Error finding container 9ba97e68bd6f6a4b8ad27817f93950c1321d8efba45ef3535a17cf020d60cbb0: Status 404 returned error can't find the container with id 9ba97e68bd6f6a4b8ad27817f93950c1321d8efba45ef3535a17cf020d60cbb0
Sep 17 23:30:00.461383 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.461361    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"]
Sep 17 23:30:00.462379 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:30:00.462359    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d0c886_813b_4ed8_abc8_0ebd45fb154d.slice/crio-2d965e69d053d99af10c659d1c39b04a4172241c84a153e09b636c2b15884ffd WatchSource:0}: Error finding container 2d965e69d053d99af10c659d1c39b04a4172241c84a153e09b636c2b15884ffd: Status 404 returned error can't find the container with id 2d965e69d053d99af10c659d1c39b04a4172241c84a153e09b636c2b15884ffd
Sep 17 23:30:00.503313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.503272    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"]
Sep 17 23:30:00.505427 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:30:00.505402    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4625ab98_a6fc_4033_b18c_77ba0611ef46.slice/crio-9789b5d81f384b3163197e4f6397f0fbe261a10b478396c5b38dd498a3534abb WatchSource:0}: Error finding container 9789b5d81f384b3163197e4f6397f0fbe261a10b478396c5b38dd498a3534abb: Status 404 returned error can't find the container with id 9789b5d81f384b3163197e4f6397f0fbe261a10b478396c5b38dd498a3534abb
Sep 17 23:30:00.521420 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.521387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48" event={"ID":"4625ab98-a6fc-4033-b18c-77ba0611ef46","Type":"ContainerStarted","Data":"9789b5d81f384b3163197e4f6397f0fbe261a10b478396c5b38dd498a3534abb"}
Sep 17 23:30:00.522790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.522759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp" event={"ID":"26631a94-c281-47c2-a9cd-47cfed5c0f71","Type":"ContainerStarted","Data":"9ba97e68bd6f6a4b8ad27817f93950c1321d8efba45ef3535a17cf020d60cbb0"}
Sep 17 23:30:00.524062 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.524040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd" event={"ID":"66d0c886-813b-4ed8-abc8-0ebd45fb154d","Type":"ContainerStarted","Data":"2d965e69d053d99af10c659d1c39b04a4172241c84a153e09b636c2b15884ffd"}
Sep 17 23:30:00.535852 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:00.535830    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:30:01.531116 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:01.531084    2559 generic.go:356] "Generic (PLEG): container finished" podID="66d0c886-813b-4ed8-abc8-0ebd45fb154d" containerID="9e4a8c98454cf0d209e89039b98de75655aaf748fd0994f1bb7c70a9172601a2" exitCode=0
Sep 17 23:30:01.531474 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:01.531170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd" event={"ID":"66d0c886-813b-4ed8-abc8-0ebd45fb154d","Type":"ContainerDied","Data":"9e4a8c98454cf0d209e89039b98de75655aaf748fd0994f1bb7c70a9172601a2"}
Sep 17 23:30:01.532787 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:01.532763    2559 generic.go:356] "Generic (PLEG): container finished" podID="4625ab98-a6fc-4033-b18c-77ba0611ef46" containerID="bc777696e778dee342f64609b98b62ad4f2e44120e35132623b07c615c1d8514" exitCode=0
Sep 17 23:30:01.532877 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:01.532831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48" event={"ID":"4625ab98-a6fc-4033-b18c-77ba0611ef46","Type":"ContainerDied","Data":"bc777696e778dee342f64609b98b62ad4f2e44120e35132623b07c615c1d8514"}
Sep 17 23:30:01.534258 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:01.534238    2559 generic.go:356] "Generic (PLEG): container finished" podID="26631a94-c281-47c2-a9cd-47cfed5c0f71" containerID="5990eec413c53bfd2bebf64cc791301560727b04662b816f8fbb06cf7d08332d" exitCode=0
Sep 17 23:30:01.534374 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:01.534304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp" event={"ID":"26631a94-c281-47c2-a9cd-47cfed5c0f71","Type":"ContainerDied","Data":"5990eec413c53bfd2bebf64cc791301560727b04662b816f8fbb06cf7d08332d"}
Sep 17 23:30:02.679096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:02.679070    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"
Sep 17 23:30:02.713121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:02.713094    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"
Sep 17 23:30:02.746004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:02.745978    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"
Sep 17 23:30:03.422693 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.422662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:30:03.546420 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.546392    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"
Sep 17 23:30:03.546592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.546391    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48" event={"ID":"4625ab98-a6fc-4033-b18c-77ba0611ef46","Type":"ContainerDied","Data":"9789b5d81f384b3163197e4f6397f0fbe261a10b478396c5b38dd498a3534abb"}
Sep 17 23:30:03.546592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.546518    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9789b5d81f384b3163197e4f6397f0fbe261a10b478396c5b38dd498a3534abb"
Sep 17 23:30:03.547959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.547941    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"
Sep 17 23:30:03.548070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.547990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp" event={"ID":"26631a94-c281-47c2-a9cd-47cfed5c0f71","Type":"ContainerDied","Data":"9ba97e68bd6f6a4b8ad27817f93950c1321d8efba45ef3535a17cf020d60cbb0"}
Sep 17 23:30:03.548070 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.548028    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ba97e68bd6f6a4b8ad27817f93950c1321d8efba45ef3535a17cf020d60cbb0"
Sep 17 23:30:03.549364 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.549341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd" event={"ID":"66d0c886-813b-4ed8-abc8-0ebd45fb154d","Type":"ContainerDied","Data":"2d965e69d053d99af10c659d1c39b04a4172241c84a153e09b636c2b15884ffd"}
Sep 17 23:30:03.549448 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.549376    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d965e69d053d99af10c659d1c39b04a4172241c84a153e09b636c2b15884ffd"
Sep 17 23:30:03.549448 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:03.549380    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"
Sep 17 23:30:09.354210 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.354179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"]
Sep 17 23:30:09.354675 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.354382    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m" podUID="707d1d6a-4be5-4765-b8c8-59cfe512fd05" containerName="payload-processing" containerID="cri-o://cf7dc56d6a6c1db635891b081153bb02648a486b4f59c05bce5e78c4a918a314" gracePeriod=30
Sep 17 23:30:09.363083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.363016    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"]
Sep 17 23:30:09.363294 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.363256    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh" podUID="8bd5ce5d-b560-4f96-a699-7e839c693d09" containerName="payload-pre-processing" containerID="cri-o://b72151e83f22c0487a841ef2701ae1a0967fe7194b5b229f259dabc18e57e6cf" gracePeriod=30
Sep 17 23:30:09.587132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.587103    2559 generic.go:356] "Generic (PLEG): container finished" podID="8bd5ce5d-b560-4f96-a699-7e839c693d09" containerID="b72151e83f22c0487a841ef2701ae1a0967fe7194b5b229f259dabc18e57e6cf" exitCode=0
Sep 17 23:30:09.587291 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.587181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh" event={"ID":"8bd5ce5d-b560-4f96-a699-7e839c693d09","Type":"ContainerDied","Data":"b72151e83f22c0487a841ef2701ae1a0967fe7194b5b229f259dabc18e57e6cf"}
Sep 17 23:30:09.589230 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.589203    2559 generic.go:356] "Generic (PLEG): container finished" podID="707d1d6a-4be5-4765-b8c8-59cfe512fd05" containerID="cf7dc56d6a6c1db635891b081153bb02648a486b4f59c05bce5e78c4a918a314" exitCode=0
Sep 17 23:30:09.589364 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.589252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m" event={"ID":"707d1d6a-4be5-4765-b8c8-59cfe512fd05","Type":"ContainerDied","Data":"cf7dc56d6a6c1db635891b081153bb02648a486b4f59c05bce5e78c4a918a314"}
Sep 17 23:30:09.639610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.639582    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:30:09.643534 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.643514    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:30:09.722885 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.722854    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707d1d6a-4be5-4765-b8c8-59cfe512fd05-plugins-config-volume\") pod \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") "
Sep 17 23:30:09.723035 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.722919    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f5ll\" (UniqueName: \"kubernetes.io/projected/8bd5ce5d-b560-4f96-a699-7e839c693d09-kube-api-access-8f5ll\") pod \"8bd5ce5d-b560-4f96-a699-7e839c693d09\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") "
Sep 17 23:30:09.723035 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.722998    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8v4m\" (UniqueName: \"kubernetes.io/projected/707d1d6a-4be5-4765-b8c8-59cfe512fd05-kube-api-access-v8v4m\") pod \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\" (UID: \"707d1d6a-4be5-4765-b8c8-59cfe512fd05\") "
Sep 17 23:30:09.723133 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.723116    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5ce5d-b560-4f96-a699-7e839c693d09-plugins-config-volume\") pod \"8bd5ce5d-b560-4f96-a699-7e839c693d09\" (UID: \"8bd5ce5d-b560-4f96-a699-7e839c693d09\") "
Sep 17 23:30:09.723262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.723239    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707d1d6a-4be5-4765-b8c8-59cfe512fd05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "707d1d6a-4be5-4765-b8c8-59cfe512fd05" (UID: "707d1d6a-4be5-4765-b8c8-59cfe512fd05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:09.723530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.723501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd5ce5d-b560-4f96-a699-7e839c693d09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bd5ce5d-b560-4f96-a699-7e839c693d09" (UID: "8bd5ce5d-b560-4f96-a699-7e839c693d09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:09.723530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.723509    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707d1d6a-4be5-4765-b8c8-59cfe512fd05-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:30:09.725085 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.725064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707d1d6a-4be5-4765-b8c8-59cfe512fd05-kube-api-access-v8v4m" (OuterVolumeSpecName: "kube-api-access-v8v4m") pod "707d1d6a-4be5-4765-b8c8-59cfe512fd05" (UID: "707d1d6a-4be5-4765-b8c8-59cfe512fd05"). InnerVolumeSpecName "kube-api-access-v8v4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:09.725171 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.725108    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd5ce5d-b560-4f96-a699-7e839c693d09-kube-api-access-8f5ll" (OuterVolumeSpecName: "kube-api-access-8f5ll") pod "8bd5ce5d-b560-4f96-a699-7e839c693d09" (UID: "8bd5ce5d-b560-4f96-a699-7e839c693d09"). InnerVolumeSpecName "kube-api-access-8f5ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:09.824419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.824399    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5ll\" (UniqueName: \"kubernetes.io/projected/8bd5ce5d-b560-4f96-a699-7e839c693d09-kube-api-access-8f5ll\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:30:09.824419 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.824417    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8v4m\" (UniqueName: \"kubernetes.io/projected/707d1d6a-4be5-4765-b8c8-59cfe512fd05-kube-api-access-v8v4m\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:30:09.824547 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:09.824429    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5ce5d-b560-4f96-a699-7e839c693d09-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:30:10.600061 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.600027    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"
Sep 17 23:30:10.600061 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.600046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m" event={"ID":"707d1d6a-4be5-4765-b8c8-59cfe512fd05","Type":"ContainerDied","Data":"da7c0e283a43185d075e2b44cb68231724d6cc9a056c1342f933dd33c3b76840"}
Sep 17 23:30:10.600526 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.600146    2559 scope.go:117] "RemoveContainer" containerID="cf7dc56d6a6c1db635891b081153bb02648a486b4f59c05bce5e78c4a918a314"
Sep 17 23:30:10.601669 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.601641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh" event={"ID":"8bd5ce5d-b560-4f96-a699-7e839c693d09","Type":"ContainerDied","Data":"d6ad6ff5f64d7baba118d49b181e14003d855a55964bd71ac5ce5001f22310a0"}
Sep 17 23:30:10.601804 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.601682    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"
Sep 17 23:30:10.613935 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.613417    2559 scope.go:117] "RemoveContainer" containerID="b72151e83f22c0487a841ef2701ae1a0967fe7194b5b229f259dabc18e57e6cf"
Sep 17 23:30:10.640118 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.640089    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"]
Sep 17 23:30:10.652730 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.652700    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b9e74c5-664d459df7-2457m"]
Sep 17 23:30:10.671260 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.671232    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"]
Sep 17 23:30:10.685161 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:10.685141    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b9e74c5-c47674cfd-hw5lh"]
Sep 17 23:30:11.639036 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:11.639005    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707d1d6a-4be5-4765-b8c8-59cfe512fd05" path="/var/lib/kubelet/pods/707d1d6a-4be5-4765-b8c8-59cfe512fd05/volumes"
Sep 17 23:30:11.639532 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:11.639511    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd5ce5d-b560-4f96-a699-7e839c693d09" path="/var/lib/kubelet/pods/8bd5ce5d-b560-4f96-a699-7e839c693d09/volumes"
Sep 17 23:30:18.429253 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:18.429223    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:30:34.622242 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:34.622201    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:30:50.530900 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:50.530860    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:30:59.671631 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:59.671588    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:30:59.674742 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:59.674692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:30:59.680463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:59.680436    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:30:59.683805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:30:59.683788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:31:04.527333 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:04.527291    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:31:18.731881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:18.731801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:31:34.523041 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:34.523007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:31:40.705436 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:40.705400    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"]
Sep 17 23:31:41.610284 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:41.610250    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"]
Sep 17 23:31:48.047496 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:48.047460    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:31:51.936328 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936296    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"]
Sep 17 23:31:51.936700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936688    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d0c886-813b-4ed8-abc8-0ebd45fb154d" containerName="cleanup"
Sep 17 23:31:51.936700 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936699    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d0c886-813b-4ed8-abc8-0ebd45fb154d" containerName="cleanup"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936712    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4625ab98-a6fc-4033-b18c-77ba0611ef46" containerName="cleanup"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936731    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4625ab98-a6fc-4033-b18c-77ba0611ef46" containerName="cleanup"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936745    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd5ce5d-b560-4f96-a699-7e839c693d09" containerName="payload-pre-processing"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936755    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd5ce5d-b560-4f96-a699-7e839c693d09" containerName="payload-pre-processing"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936782    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707d1d6a-4be5-4765-b8c8-59cfe512fd05" containerName="payload-processing"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936788    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="707d1d6a-4be5-4765-b8c8-59cfe512fd05" containerName="payload-processing"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936800    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26631a94-c281-47c2-a9cd-47cfed5c0f71" containerName="cleanup"
Sep 17 23:31:51.936838 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936806    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="26631a94-c281-47c2-a9cd-47cfed5c0f71" containerName="cleanup"
Sep 17 23:31:51.937132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936869    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd5ce5d-b560-4f96-a699-7e839c693d09" containerName="payload-pre-processing"
Sep 17 23:31:51.937132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936878    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4625ab98-a6fc-4033-b18c-77ba0611ef46" containerName="cleanup"
Sep 17 23:31:51.937132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936885    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d0c886-813b-4ed8-abc8-0ebd45fb154d" containerName="cleanup"
Sep 17 23:31:51.937132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936892    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="26631a94-c281-47c2-a9cd-47cfed5c0f71" containerName="cleanup"
Sep 17 23:31:51.937132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.936897    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="707d1d6a-4be5-4765-b8c8-59cfe512fd05" containerName="payload-processing"
Sep 17 23:31:51.940028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.940012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:51.942388 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.942366    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-78432f-dockercfg-ttpbl\""
Sep 17 23:31:51.942501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.942366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-78432f\""
Sep 17 23:31:51.950380 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:51.950353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"]
Sep 17 23:31:52.023131 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.023092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.023263 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.023156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btf7p\" (UniqueName: \"kubernetes.io/projected/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-kube-api-access-btf7p\") pod \"payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.124231 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.124201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btf7p\" (UniqueName: \"kubernetes.io/projected/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-kube-api-access-btf7p\") pod \"payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.124354 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.124291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.124959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.124941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.131463 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.131444    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btf7p\" (UniqueName: \"kubernetes.io/projected/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-kube-api-access-btf7p\") pod \"payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.251625 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.251604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:52.526537 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:52.523883    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"]
Sep 17 23:31:52.527614 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:31:52.527582    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ccefb5_ba91_41b6_8b46_8e2aaedbf055.slice/crio-76c24582aa73bc984854a121f99ba680d327d6bf2a7df02becc912a6ddc748e9 WatchSource:0}: Error finding container 76c24582aa73bc984854a121f99ba680d327d6bf2a7df02becc912a6ddc748e9: Status 404 returned error can't find the container with id 76c24582aa73bc984854a121f99ba680d327d6bf2a7df02becc912a6ddc748e9
Sep 17 23:31:53.180429 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:53.180394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z" event={"ID":"45ccefb5-ba91-41b6-8b46-8e2aaedbf055","Type":"ContainerStarted","Data":"ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271"}
Sep 17 23:31:53.180839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:53.180449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z" event={"ID":"45ccefb5-ba91-41b6-8b46-8e2aaedbf055","Type":"ContainerStarted","Data":"76c24582aa73bc984854a121f99ba680d327d6bf2a7df02becc912a6ddc748e9"}
Sep 17 23:31:53.180839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:53.180545    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:31:53.197276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:53.197231    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z" podStartSLOduration=2.197218509 podStartE2EDuration="2.197218509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:31:53.194448018 +0000 UTC m=+1555.668391001" watchObservedRunningTime="2026-09-17 23:31:53.197218509 +0000 UTC m=+1555.671161524"
Sep 17 23:31:55.727373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:55.727330    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9" podUID="d9ee0094-1e9d-4677-8c27-d153ea5af8af" containerName="main" containerID="cri-o://c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8" gracePeriod=45
Sep 17 23:31:55.981551 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:55.981498    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:31:56.059948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.059912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh4cs\" (UniqueName: \"kubernetes.io/projected/d9ee0094-1e9d-4677-8c27-d153ea5af8af-kube-api-access-lh4cs\") pod \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") "
Sep 17 23:31:56.060096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.059984    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-model-cache\") pod \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") "
Sep 17 23:31:56.060096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060020    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-dshm\") pod \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") "
Sep 17 23:31:56.060096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060056    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tmp-dir\") pod \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") "
Sep 17 23:31:56.060096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060093    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-home\") pod \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") "
Sep 17 23:31:56.060268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060162    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tls-certs\") pod \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\" (UID: \"d9ee0094-1e9d-4677-8c27-d153ea5af8af\") "
Sep 17 23:31:56.060324 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060292    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9ee0094-1e9d-4677-8c27-d153ea5af8af" (UID: "d9ee0094-1e9d-4677-8c27-d153ea5af8af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.060386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9ee0094-1e9d-4677-8c27-d153ea5af8af" (UID: "d9ee0094-1e9d-4677-8c27-d153ea5af8af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.060439 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060387    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-home" (OuterVolumeSpecName: "home") pod "d9ee0094-1e9d-4677-8c27-d153ea5af8af" (UID: "d9ee0094-1e9d-4677-8c27-d153ea5af8af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.060539 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060523    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.060590 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060541    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.060590 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.060551    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.062270 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.062249    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-dshm" (OuterVolumeSpecName: "dshm") pod "d9ee0094-1e9d-4677-8c27-d153ea5af8af" (UID: "d9ee0094-1e9d-4677-8c27-d153ea5af8af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.062361 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.062276    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9ee0094-1e9d-4677-8c27-d153ea5af8af" (UID: "d9ee0094-1e9d-4677-8c27-d153ea5af8af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:56.062401 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.062371    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ee0094-1e9d-4677-8c27-d153ea5af8af-kube-api-access-lh4cs" (OuterVolumeSpecName: "kube-api-access-lh4cs") pod "d9ee0094-1e9d-4677-8c27-d153ea5af8af" (UID: "d9ee0094-1e9d-4677-8c27-d153ea5af8af"). InnerVolumeSpecName "kube-api-access-lh4cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:56.161998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.161972    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ee0094-1e9d-4677-8c27-d153ea5af8af-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.161998 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.161995    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4cs\" (UniqueName: \"kubernetes.io/projected/d9ee0094-1e9d-4677-8c27-d153ea5af8af-kube-api-access-lh4cs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.162137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.162005    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ee0094-1e9d-4677-8c27-d153ea5af8af-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.198349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.198322    2559 generic.go:356] "Generic (PLEG): container finished" podID="d9ee0094-1e9d-4677-8c27-d153ea5af8af" containerID="c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8" exitCode=0
Sep 17 23:31:56.198485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.198362    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9" event={"ID":"d9ee0094-1e9d-4677-8c27-d153ea5af8af","Type":"ContainerDied","Data":"c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8"}
Sep 17 23:31:56.198485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.198373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"
Sep 17 23:31:56.198485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.198389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9" event={"ID":"d9ee0094-1e9d-4677-8c27-d153ea5af8af","Type":"ContainerDied","Data":"5bd02abf30ba3174770fc833ca930f92917f4f8c424267becd9d87f0d28fe74c"}
Sep 17 23:31:56.198485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.198405    2559 scope.go:117] "RemoveContainer" containerID="c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8"
Sep 17 23:31:56.207598 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.207581    2559 scope.go:117] "RemoveContainer" containerID="c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8"
Sep 17 23:31:56.207874 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:31:56.207854    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8\": container with ID starting with c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8 not found: ID does not exist" containerID="c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8"
Sep 17 23:31:56.207937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.207882    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8"} err="failed to get container status \"c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8\": rpc error: code = NotFound desc = could not find container \"c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8\": container with ID starting with c7fb3f1eb07b8b88d385b691101f35fe98fc57acc2f8cf2faf7a2966bb0e71c8 not found: ID does not exist"
Sep 17 23:31:56.222890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.222862    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"]
Sep 17 23:31:56.228650 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.228630    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/auth-test-model-d90df4-kserve-585b45db55-267l9"]
Sep 17 23:31:56.629948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.629879    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn" podUID="5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" containerName="main" containerID="cri-o://61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a" gracePeriod=45
Sep 17 23:31:56.877296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.877273    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:31:56.969952 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.969925    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-dshm\") pod \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") "
Sep 17 23:31:56.970083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.969991    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-model-cache\") pod \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") "
Sep 17 23:31:56.970083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970007    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tmp-dir\") pod \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") "
Sep 17 23:31:56.970083 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970034    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-home\") pod \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") "
Sep 17 23:31:56.970236 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970082    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvvj9\" (UniqueName: \"kubernetes.io/projected/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-kube-api-access-xvvj9\") pod \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") "
Sep 17 23:31:56.970236 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970164    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tls-certs\") pod \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\" (UID: \"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c\") "
Sep 17 23:31:56.970391 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970274    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" (UID: "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.970391 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" (UID: "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.970391 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970300    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-home" (OuterVolumeSpecName: "home") pod "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" (UID: "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.970541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970449    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.970541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970473    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.970541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.970480    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.972260 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.972229    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-dshm" (OuterVolumeSpecName: "dshm") pod "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" (UID: "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.972373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.972263    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-kube-api-access-xvvj9" (OuterVolumeSpecName: "kube-api-access-xvvj9") pod "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" (UID: "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c"). InnerVolumeSpecName "kube-api-access-xvvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:56.972373 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:56.972281    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" (UID: "5fd8d4c1-5c23-4b23-b681-a53f487d0b9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:57.071792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.071764    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvj9\" (UniqueName: \"kubernetes.io/projected/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-kube-api-access-xvvj9\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.071792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.071788    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.071924 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.071801    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.206271 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.206235    2559 generic.go:356] "Generic (PLEG): container finished" podID="5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" containerID="61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a" exitCode=0
Sep 17 23:31:57.206410 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.206277    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"
Sep 17 23:31:57.206410 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.206284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn" event={"ID":"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c","Type":"ContainerDied","Data":"61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a"}
Sep 17 23:31:57.206410 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.206321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn" event={"ID":"5fd8d4c1-5c23-4b23-b681-a53f487d0b9c","Type":"ContainerDied","Data":"33f015b96cf204d3012665be34925a8b9ebc26ede6387c51a3314af052c03722"}
Sep 17 23:31:57.206410 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.206344    2559 scope.go:117] "RemoveContainer" containerID="61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a"
Sep 17 23:31:57.219468 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.219365    2559 scope.go:117] "RemoveContainer" containerID="61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a"
Sep 17 23:31:57.220348 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:31:57.219777    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a\": container with ID starting with 61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a not found: ID does not exist" containerID="61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a"
Sep 17 23:31:57.220348 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.219817    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a"} err="failed to get container status \"61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a\": rpc error: code = NotFound desc = could not find container \"61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a\": container with ID starting with 61d1bddab6bd7ade04e239b956687cfd22775c19568127dede05c1e4f3f0447a not found: ID does not exist"
Sep 17 23:31:57.233110 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.233088    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"]
Sep 17 23:31:57.234975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.234956    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/auth-test-model-d92920-kserve-69b84ccc47-2wxdn"]
Sep 17 23:31:57.639881 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.639812    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" path="/var/lib/kubelet/pods/5fd8d4c1-5c23-4b23-b681-a53f487d0b9c/volumes"
Sep 17 23:31:57.640258 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:57.640239    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ee0094-1e9d-4677-8c27-d153ea5af8af" path="/var/lib/kubelet/pods/d9ee0094-1e9d-4677-8c27-d153ea5af8af/volumes"
Sep 17 23:31:59.813484 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:31:59.813452    2559 scope.go:117] "RemoveContainer" containerID="1972259ab01841174bd7c93be7bb83ccc11cda256afd269a524c85495b10ee7a"
Sep 17 23:32:02.345753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:02.345709    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:32:04.187586 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:04.187553    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:32:08.047848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.047803    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"]
Sep 17 23:32:08.048291 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.048274    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ee0094-1e9d-4677-8c27-d153ea5af8af" containerName="main"
Sep 17 23:32:08.048291 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.048291    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ee0094-1e9d-4677-8c27-d153ea5af8af" containerName="main"
Sep 17 23:32:08.048385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.048300    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" containerName="main"
Sep 17 23:32:08.048385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.048307    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" containerName="main"
Sep 17 23:32:08.048452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.048397    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd8d4c1-5c23-4b23-b681-a53f487d0b9c" containerName="main"
Sep 17 23:32:08.048452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.048410    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ee0094-1e9d-4677-8c27-d153ea5af8af" containerName="main"
Sep 17 23:32:08.053256 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.053232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.056594 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.056560    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"openshift-service-ca.crt\""
Sep 17 23:32:08.056740 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.056617    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"sub-test-model-d90df4-kserve-self-signed-certs\""
Sep 17 23:32:08.056740 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.056651    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"default-dockercfg-6zl8d\""
Sep 17 23:32:08.056879 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.056783    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"kube-root-ca.crt\""
Sep 17 23:32:08.061523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.061497    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"]
Sep 17 23:32:08.168811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.168780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-home\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.168913 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.168833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-dshm\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.168913 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.168859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2dv\" (UniqueName: \"kubernetes.io/projected/86d5bf60-3451-472f-884d-503c31c32b1a-kube-api-access-gf2dv\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.168913 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.168887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-model-cache\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.168913 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.168908    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-tmp-dir\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.169059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.168943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d5bf60-3451-472f-884d-503c31c32b1a-tls-certs\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270250 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-home\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-dshm\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2dv\" (UniqueName: \"kubernetes.io/projected/86d5bf60-3451-472f-884d-503c31c32b1a-kube-api-access-gf2dv\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-model-cache\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-tmp-dir\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270580 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d5bf60-3451-472f-884d-503c31c32b1a-tls-certs\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270629 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-home\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270817 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-model-cache\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.270898 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.270815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-tmp-dir\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.272675 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.272650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-dshm\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.273012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.272995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d5bf60-3451-472f-884d-503c31c32b1a-tls-certs\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.277592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.277574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2dv\" (UniqueName: \"kubernetes.io/projected/86d5bf60-3451-472f-884d-503c31c32b1a-kube-api-access-gf2dv\") pod \"sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.365239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.365179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:08.637941 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:08.637897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"]
Sep 17 23:32:08.644630 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:32:08.644590    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86d5bf60_3451_472f_884d_503c31c32b1a.slice/crio-46aa2c6d654b88768d86997956861d1d47afacc1e7a474198b75f2166576544d WatchSource:0}: Error finding container 46aa2c6d654b88768d86997956861d1d47afacc1e7a474198b75f2166576544d: Status 404 returned error can't find the container with id 46aa2c6d654b88768d86997956861d1d47afacc1e7a474198b75f2166576544d
Sep 17 23:32:09.279631 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:09.279596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m" event={"ID":"86d5bf60-3451-472f-884d-503c31c32b1a","Type":"ContainerStarted","Data":"a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42"}
Sep 17 23:32:09.280038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:09.279643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m" event={"ID":"86d5bf60-3451-472f-884d-503c31c32b1a","Type":"ContainerStarted","Data":"46aa2c6d654b88768d86997956861d1d47afacc1e7a474198b75f2166576544d"}
Sep 17 23:32:09.298332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:09.298283    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m" podStartSLOduration=1.298267376 podStartE2EDuration="1.298267376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:09.29675857 +0000 UTC m=+1571.770701554" watchObservedRunningTime="2026-09-17 23:32:09.298267376 +0000 UTC m=+1571.772210425"
Sep 17 23:32:15.534573 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:15.534539    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:32:18.366106 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:18.366049    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:18.366556 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:18.366130    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:18.378939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:18.378912    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:19.350597 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:19.350564    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:32:26.348350 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.348319    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"]
Sep 17 23:32:26.352156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.352135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.357290 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.357261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cf8993\""
Sep 17 23:32:26.357969 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.357947    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cf8993-dockercfg-lmjxb\""
Sep 17 23:32:26.359683 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.359662    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"]
Sep 17 23:32:26.362889 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.362872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.368361 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.368342    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"]
Sep 17 23:32:26.378327 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.378307    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"]
Sep 17 23:32:26.421435 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.421408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf8993-567598db9-wsf8t\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.421560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.421456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59hvp\" (UniqueName: \"kubernetes.io/projected/112edc5d-3266-427a-92fe-bede93d8feed-kube-api-access-59hvp\") pod \"payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.421560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.421549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112edc5d-3266-427a-92fe-bede93d8feed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.421677 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.421580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxh2d\" (UniqueName: \"kubernetes.io/projected/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-kube-api-access-cxh2d\") pod \"payload-processing-e2e-api-b-cf8993-567598db9-wsf8t\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.522594 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.522564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf8993-567598db9-wsf8t\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.522750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.522605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59hvp\" (UniqueName: \"kubernetes.io/projected/112edc5d-3266-427a-92fe-bede93d8feed-kube-api-access-59hvp\") pod \"payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.522750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.522642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112edc5d-3266-427a-92fe-bede93d8feed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.522750 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.522669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxh2d\" (UniqueName: \"kubernetes.io/projected/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-kube-api-access-cxh2d\") pod \"payload-processing-e2e-api-b-cf8993-567598db9-wsf8t\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.523362 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.523344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf8993-567598db9-wsf8t\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.523447 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.523429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112edc5d-3266-427a-92fe-bede93d8feed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.530254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.530232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxh2d\" (UniqueName: \"kubernetes.io/projected/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-kube-api-access-cxh2d\") pod \"payload-processing-e2e-api-b-cf8993-567598db9-wsf8t\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.530347 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.530327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59hvp\" (UniqueName: \"kubernetes.io/projected/112edc5d-3266-427a-92fe-bede93d8feed-kube-api-access-59hvp\") pod \"payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.662866 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.662808    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:26.673310 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.673291    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:26.940899 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.940867    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"]
Sep 17 23:32:26.942977 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:32:26.942944    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112edc5d_3266_427a_92fe_bede93d8feed.slice/crio-f872571849cb9f77b6f4632b94e81e4080f2fe96f214e48155881470c59d84c0 WatchSource:0}: Error finding container f872571849cb9f77b6f4632b94e81e4080f2fe96f214e48155881470c59d84c0: Status 404 returned error can't find the container with id f872571849cb9f77b6f4632b94e81e4080f2fe96f214e48155881470c59d84c0
Sep 17 23:32:26.959318 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:26.959292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"]
Sep 17 23:32:26.960511 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:32:26.960488    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664c3663_c9d3_4af1_b1e7_d39b8a5422f3.slice/crio-2defc551f6b1506504e69d66d46dfeebd60aeb6fe245baad5f8b8f63a2d103f7 WatchSource:0}: Error finding container 2defc551f6b1506504e69d66d46dfeebd60aeb6fe245baad5f8b8f63a2d103f7: Status 404 returned error can't find the container with id 2defc551f6b1506504e69d66d46dfeebd60aeb6fe245baad5f8b8f63a2d103f7
Sep 17 23:32:27.397237 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.397198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t" event={"ID":"664c3663-c9d3-4af1-b1e7-d39b8a5422f3","Type":"ContainerStarted","Data":"d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77"}
Sep 17 23:32:27.397636 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.397267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t" event={"ID":"664c3663-c9d3-4af1-b1e7-d39b8a5422f3","Type":"ContainerStarted","Data":"2defc551f6b1506504e69d66d46dfeebd60aeb6fe245baad5f8b8f63a2d103f7"}
Sep 17 23:32:27.397636 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.397296    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:27.398780 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.398754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z" event={"ID":"112edc5d-3266-427a-92fe-bede93d8feed","Type":"ContainerStarted","Data":"d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa"}
Sep 17 23:32:27.398903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.398795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z" event={"ID":"112edc5d-3266-427a-92fe-bede93d8feed","Type":"ContainerStarted","Data":"f872571849cb9f77b6f4632b94e81e4080f2fe96f214e48155881470c59d84c0"}
Sep 17 23:32:27.398903 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.398835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:27.413586 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.413548    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t" podStartSLOduration=1.41353181 podStartE2EDuration="1.41353181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:27.411014554 +0000 UTC m=+1589.884957540" watchObservedRunningTime="2026-09-17 23:32:27.41353181 +0000 UTC m=+1589.887474796"
Sep 17 23:32:27.428284 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:27.428245    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z" podStartSLOduration=1.428232908 podStartE2EDuration="1.428232908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:27.425714177 +0000 UTC m=+1589.899657161" watchObservedRunningTime="2026-09-17 23:32:27.428232908 +0000 UTC m=+1589.902175940"
Sep 17 23:32:33.433822 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:33.433782    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:32:37.828425 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.828382    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"]
Sep 17 23:32:37.834825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.834793    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:37.837411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.837386    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"default-dockercfg-mwr9r\""
Sep 17 23:32:37.838480 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.838446    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"sub-test-model-d92920-kserve-self-signed-certs\""
Sep 17 23:32:37.838930 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.838909    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"kube-root-ca.crt\""
Sep 17 23:32:37.839079 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.839058    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d92920\"/\"openshift-service-ca.crt\""
Sep 17 23:32:37.843193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.843167    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"]
Sep 17 23:32:37.919920 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.919888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tls-certs\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:37.920028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.919921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-model-cache\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:37.920028 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.919971    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-home\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:37.920145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.920025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tmp-dir\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:37.920145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.920083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-dshm\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:37.920145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:37.920123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2hhf\" (UniqueName: \"kubernetes.io/projected/b9a79ed4-659c-4bc6-a103-29b332eb4edd-kube-api-access-l2hhf\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tls-certs\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021149 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-model-cache\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021149 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-home\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021226 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tmp-dir\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021316 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-dshm\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021369 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2hhf\" (UniqueName: \"kubernetes.io/projected/b9a79ed4-659c-4bc6-a103-29b332eb4edd-kube-api-access-l2hhf\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021606 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-model-cache\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021606 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tmp-dir\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.021757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.021666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-home\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.023778 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.023749    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-dshm\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.023866 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.023823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tls-certs\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.028567 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.028540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2hhf\" (UniqueName: \"kubernetes.io/projected/b9a79ed4-659c-4bc6-a103-29b332eb4edd-kube-api-access-l2hhf\") pod \"sub-test-model-d92920-kserve-759967d7cf-v4n2x\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") " pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.151160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.151096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:38.405967 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.405895    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:32:38.406091 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.405977    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:32:38.426327 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.426305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"]
Sep 17 23:32:38.428659 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:32:38.428629    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9a79ed4_659c_4bc6_a103_29b332eb4edd.slice/crio-17e396a734966ac66ff8245d86c1277068d5f1e46f15b725c21fbedd884256d1 WatchSource:0}: Error finding container 17e396a734966ac66ff8245d86c1277068d5f1e46f15b725c21fbedd884256d1: Status 404 returned error can't find the container with id 17e396a734966ac66ff8245d86c1277068d5f1e46f15b725c21fbedd884256d1
Sep 17 23:32:38.463846 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:38.463812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x" event={"ID":"b9a79ed4-659c-4bc6-a103-29b332eb4edd","Type":"ContainerStarted","Data":"17e396a734966ac66ff8245d86c1277068d5f1e46f15b725c21fbedd884256d1"}
Sep 17 23:32:39.471986 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:39.471943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x" event={"ID":"b9a79ed4-659c-4bc6-a103-29b332eb4edd","Type":"ContainerStarted","Data":"5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa"}
Sep 17 23:32:39.488200 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:39.488148    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x" podStartSLOduration=2.488130258 podStartE2EDuration="2.488130258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:39.486636963 +0000 UTC m=+1601.960579948" watchObservedRunningTime="2026-09-17 23:32:39.488130258 +0000 UTC m=+1601.962073240"
Sep 17 23:32:48.151855 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:48.151763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:48.151855 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:48.151835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:48.164632 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:48.164608    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:48.535335 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:48.535303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:32:57.323168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:57.323135    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:32:59.841874 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:59.841838    2559 scope.go:117] "RemoveContainer" containerID="13c45e85e8ed9c9232ea286a1c86d132ca99a81f7d03231cf0adab6223449f70"
Sep 17 23:32:59.850537 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:32:59.850514    2559 scope.go:117] "RemoveContainer" containerID="8c93561718574abcc60facd2d6301ba2779e5bc8f3a3b4a5176ca62ceb562585"
Sep 17 23:33:10.310702 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.310667    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"]
Sep 17 23:33:10.311191 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.310952    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z" podUID="45ccefb5-ba91-41b6-8b46-8e2aaedbf055" containerName="payload-pre-processing" containerID="cri-o://ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271" gracePeriod=30
Sep 17 23:33:10.569226 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.569170    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:33:10.627868 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.627841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btf7p\" (UniqueName: \"kubernetes.io/projected/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-kube-api-access-btf7p\") pod \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") "
Sep 17 23:33:10.627987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.627880    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-plugins-config-volume\") pod \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\" (UID: \"45ccefb5-ba91-41b6-8b46-8e2aaedbf055\") "
Sep 17 23:33:10.628259 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.628239    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45ccefb5-ba91-41b6-8b46-8e2aaedbf055" (UID: "45ccefb5-ba91-41b6-8b46-8e2aaedbf055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:10.629968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.629942    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-kube-api-access-btf7p" (OuterVolumeSpecName: "kube-api-access-btf7p") pod "45ccefb5-ba91-41b6-8b46-8e2aaedbf055" (UID: "45ccefb5-ba91-41b6-8b46-8e2aaedbf055"). InnerVolumeSpecName "kube-api-access-btf7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:10.649470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.649437    2559 generic.go:356] "Generic (PLEG): container finished" podID="45ccefb5-ba91-41b6-8b46-8e2aaedbf055" containerID="ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271" exitCode=0
Sep 17 23:33:10.649551 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.649478    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"
Sep 17 23:33:10.649551 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.649482    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z" event={"ID":"45ccefb5-ba91-41b6-8b46-8e2aaedbf055","Type":"ContainerDied","Data":"ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271"}
Sep 17 23:33:10.649551 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.649528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z" event={"ID":"45ccefb5-ba91-41b6-8b46-8e2aaedbf055","Type":"ContainerDied","Data":"76c24582aa73bc984854a121f99ba680d327d6bf2a7df02becc912a6ddc748e9"}
Sep 17 23:33:10.649660 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.649558    2559 scope.go:117] "RemoveContainer" containerID="ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271"
Sep 17 23:33:10.658555 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.658539    2559 scope.go:117] "RemoveContainer" containerID="ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271"
Sep 17 23:33:10.658827 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:33:10.658807    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271\": container with ID starting with ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271 not found: ID does not exist" containerID="ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271"
Sep 17 23:33:10.658879 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.658835    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271"} err="failed to get container status \"ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271\": rpc error: code = NotFound desc = could not find container \"ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271\": container with ID starting with ad548cef3a384cb756fe9bf7f82a96d8d6c336efb0f1aca2489f75c0a10f2271 not found: ID does not exist"
Sep 17 23:33:10.670310 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.670291    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"]
Sep 17 23:33:10.673974 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.673953    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78432f-77657d5bdb-66g7z"]
Sep 17 23:33:10.729288 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.729261    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btf7p\" (UniqueName: \"kubernetes.io/projected/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-kube-api-access-btf7p\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:10.729411 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:10.729399    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccefb5-ba91-41b6-8b46-8e2aaedbf055-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:11.639669 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:11.639632    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ccefb5-ba91-41b6-8b46-8e2aaedbf055" path="/var/lib/kubelet/pods/45ccefb5-ba91-41b6-8b46-8e2aaedbf055/volumes"
Sep 17 23:33:12.232773 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:12.232742    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:33:28.726474 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:28.726420    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:33:30.919959 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:30.919920    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"]
Sep 17 23:33:32.017366 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:32.017334    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"]
Sep 17 23:33:32.853842 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:32.853811    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"]
Sep 17 23:33:32.854089 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:32.854039    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t" podUID="664c3663-c9d3-4af1-b1e7-d39b8a5422f3" containerName="payload-processing" containerID="cri-o://d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77" gracePeriod=30
Sep 17 23:33:32.860927 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:32.860901    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"]
Sep 17 23:33:32.861132 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:32.861110    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z" podUID="112edc5d-3266-427a-92fe-bede93d8feed" containerName="payload-pre-processing" containerID="cri-o://d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa" gracePeriod=30
Sep 17 23:33:33.131749 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.131710    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:33:33.134564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.134544    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:33:33.218566 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.218528    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxh2d\" (UniqueName: \"kubernetes.io/projected/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-kube-api-access-cxh2d\") pod \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") "
Sep 17 23:33:33.218790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.218634    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-plugins-config-volume\") pod \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\" (UID: \"664c3663-c9d3-4af1-b1e7-d39b8a5422f3\") "
Sep 17 23:33:33.218790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.218676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59hvp\" (UniqueName: \"kubernetes.io/projected/112edc5d-3266-427a-92fe-bede93d8feed-kube-api-access-59hvp\") pod \"112edc5d-3266-427a-92fe-bede93d8feed\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") "
Sep 17 23:33:33.218790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.218770    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112edc5d-3266-427a-92fe-bede93d8feed-plugins-config-volume\") pod \"112edc5d-3266-427a-92fe-bede93d8feed\" (UID: \"112edc5d-3266-427a-92fe-bede93d8feed\") "
Sep 17 23:33:33.219122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.219090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112edc5d-3266-427a-92fe-bede93d8feed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "112edc5d-3266-427a-92fe-bede93d8feed" (UID: "112edc5d-3266-427a-92fe-bede93d8feed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:33.219229 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.219119    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "664c3663-c9d3-4af1-b1e7-d39b8a5422f3" (UID: "664c3663-c9d3-4af1-b1e7-d39b8a5422f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:33.220923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.220896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-kube-api-access-cxh2d" (OuterVolumeSpecName: "kube-api-access-cxh2d") pod "664c3663-c9d3-4af1-b1e7-d39b8a5422f3" (UID: "664c3663-c9d3-4af1-b1e7-d39b8a5422f3"). InnerVolumeSpecName "kube-api-access-cxh2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:33.220923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.220912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112edc5d-3266-427a-92fe-bede93d8feed-kube-api-access-59hvp" (OuterVolumeSpecName: "kube-api-access-59hvp") pod "112edc5d-3266-427a-92fe-bede93d8feed" (UID: "112edc5d-3266-427a-92fe-bede93d8feed"). InnerVolumeSpecName "kube-api-access-59hvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:33.320186 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.320143    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxh2d\" (UniqueName: \"kubernetes.io/projected/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-kube-api-access-cxh2d\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:33.320186 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.320177    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664c3663-c9d3-4af1-b1e7-d39b8a5422f3-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:33.320186 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.320190    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59hvp\" (UniqueName: \"kubernetes.io/projected/112edc5d-3266-427a-92fe-bede93d8feed-kube-api-access-59hvp\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:33.320431 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.320204    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112edc5d-3266-427a-92fe-bede93d8feed-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:33.779955 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.779909    2559 generic.go:356] "Generic (PLEG): container finished" podID="664c3663-c9d3-4af1-b1e7-d39b8a5422f3" containerID="d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77" exitCode=0
Sep 17 23:33:33.780156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.779969    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"
Sep 17 23:33:33.780156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.779991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t" event={"ID":"664c3663-c9d3-4af1-b1e7-d39b8a5422f3","Type":"ContainerDied","Data":"d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77"}
Sep 17 23:33:33.780156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.780047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t" event={"ID":"664c3663-c9d3-4af1-b1e7-d39b8a5422f3","Type":"ContainerDied","Data":"2defc551f6b1506504e69d66d46dfeebd60aeb6fe245baad5f8b8f63a2d103f7"}
Sep 17 23:33:33.780156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.780073    2559 scope.go:117] "RemoveContainer" containerID="d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77"
Sep 17 23:33:33.781392 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.781361    2559 generic.go:356] "Generic (PLEG): container finished" podID="112edc5d-3266-427a-92fe-bede93d8feed" containerID="d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa" exitCode=0
Sep 17 23:33:33.781523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.781422    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"
Sep 17 23:33:33.781523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.781457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z" event={"ID":"112edc5d-3266-427a-92fe-bede93d8feed","Type":"ContainerDied","Data":"d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa"}
Sep 17 23:33:33.781523 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.781496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z" event={"ID":"112edc5d-3266-427a-92fe-bede93d8feed","Type":"ContainerDied","Data":"f872571849cb9f77b6f4632b94e81e4080f2fe96f214e48155881470c59d84c0"}
Sep 17 23:33:33.792685 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.792666    2559 scope.go:117] "RemoveContainer" containerID="d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77"
Sep 17 23:33:33.792978 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:33:33.792957    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77\": container with ID starting with d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77 not found: ID does not exist" containerID="d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77"
Sep 17 23:33:33.793037 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.792982    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77"} err="failed to get container status \"d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77\": rpc error: code = NotFound desc = could not find container \"d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77\": container with ID starting with d2110ed41a063b5120d64e1913548c0ef167efd85f63eb6fc2f64a6504ebbb77 not found: ID does not exist"
Sep 17 23:33:33.793037 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.792998    2559 scope.go:117] "RemoveContainer" containerID="d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa"
Sep 17 23:33:33.796979 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.796957    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"]
Sep 17 23:33:33.801685 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.801667    2559 scope.go:117] "RemoveContainer" containerID="d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa"
Sep 17 23:33:33.802019 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:33:33.802000    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa\": container with ID starting with d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa not found: ID does not exist" containerID="d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa"
Sep 17 23:33:33.802105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.802029    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa"} err="failed to get container status \"d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa\": rpc error: code = NotFound desc = could not find container \"d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa\": container with ID starting with d47cc308058bed32dfa0a2bbbff3c14d9eafcc2575da164853207da44b30d6aa not found: ID does not exist"
Sep 17 23:33:33.803215 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.803196    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf8993-56f99d8c6b-msp5z"]
Sep 17 23:33:33.816196 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.816171    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"]
Sep 17 23:33:33.819670 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:33.819643    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf8993-567598db9-wsf8t"]
Sep 17 23:33:35.640475 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:35.640440    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="112edc5d-3266-427a-92fe-bede93d8feed" path="/var/lib/kubelet/pods/112edc5d-3266-427a-92fe-bede93d8feed/volumes"
Sep 17 23:33:35.640872 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:35.640844    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="664c3663-c9d3-4af1-b1e7-d39b8a5422f3" path="/var/lib/kubelet/pods/664c3663-c9d3-4af1-b1e7-d39b8a5422f3/volumes"
Sep 17 23:33:44.326285 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:44.326245    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:33:45.939684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:45.939617    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m" podUID="86d5bf60-3451-472f-884d-503c31c32b1a" containerName="main" containerID="cri-o://a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42" gracePeriod=45
Sep 17 23:33:46.200459 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.200438    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:33:46.346203 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346176    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf2dv\" (UniqueName: \"kubernetes.io/projected/86d5bf60-3451-472f-884d-503c31c32b1a-kube-api-access-gf2dv\") pod \"86d5bf60-3451-472f-884d-503c31c32b1a\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") "
Sep 17 23:33:46.346358 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346233    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-tmp-dir\") pod \"86d5bf60-3451-472f-884d-503c31c32b1a\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") "
Sep 17 23:33:46.346358 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346267    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-dshm\") pod \"86d5bf60-3451-472f-884d-503c31c32b1a\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") "
Sep 17 23:33:46.346358 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346316    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-home\") pod \"86d5bf60-3451-472f-884d-503c31c32b1a\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") "
Sep 17 23:33:46.346531 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-model-cache\") pod \"86d5bf60-3451-472f-884d-503c31c32b1a\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") "
Sep 17 23:33:46.346531 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346393    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d5bf60-3451-472f-884d-503c31c32b1a-tls-certs\") pod \"86d5bf60-3451-472f-884d-503c31c32b1a\" (UID: \"86d5bf60-3451-472f-884d-503c31c32b1a\") "
Sep 17 23:33:46.346641 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86d5bf60-3451-472f-884d-503c31c32b1a" (UID: "86d5bf60-3451-472f-884d-503c31c32b1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:46.346694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-home" (OuterVolumeSpecName: "home") pod "86d5bf60-3451-472f-884d-503c31c32b1a" (UID: "86d5bf60-3451-472f-884d-503c31c32b1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:46.346694 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346667    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "86d5bf60-3451-472f-884d-503c31c32b1a" (UID: "86d5bf60-3451-472f-884d-503c31c32b1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:46.346833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346814    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.346833 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346833    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.346937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.346841    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.348333 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.348308    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-dshm" (OuterVolumeSpecName: "dshm") pod "86d5bf60-3451-472f-884d-503c31c32b1a" (UID: "86d5bf60-3451-472f-884d-503c31c32b1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:46.348502 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.348482    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d5bf60-3451-472f-884d-503c31c32b1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86d5bf60-3451-472f-884d-503c31c32b1a" (UID: "86d5bf60-3451-472f-884d-503c31c32b1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:46.348550 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.348495    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d5bf60-3451-472f-884d-503c31c32b1a-kube-api-access-gf2dv" (OuterVolumeSpecName: "kube-api-access-gf2dv") pod "86d5bf60-3451-472f-884d-503c31c32b1a" (UID: "86d5bf60-3451-472f-884d-503c31c32b1a"). InnerVolumeSpecName "kube-api-access-gf2dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:46.448104 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.448084    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d5bf60-3451-472f-884d-503c31c32b1a-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.448104 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.448105    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d5bf60-3451-472f-884d-503c31c32b1a-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.448245 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.448115    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2dv\" (UniqueName: \"kubernetes.io/projected/86d5bf60-3451-472f-884d-503c31c32b1a-kube-api-access-gf2dv\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.854385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.854349    2559 generic.go:356] "Generic (PLEG): container finished" podID="86d5bf60-3451-472f-884d-503c31c32b1a" containerID="a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42" exitCode=0
Sep 17 23:33:46.854562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.854401    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"
Sep 17 23:33:46.854562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.854394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m" event={"ID":"86d5bf60-3451-472f-884d-503c31c32b1a","Type":"ContainerDied","Data":"a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42"}
Sep 17 23:33:46.854562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.854519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m" event={"ID":"86d5bf60-3451-472f-884d-503c31c32b1a","Type":"ContainerDied","Data":"46aa2c6d654b88768d86997956861d1d47afacc1e7a474198b75f2166576544d"}
Sep 17 23:33:46.854562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.854541    2559 scope.go:117] "RemoveContainer" containerID="a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42"
Sep 17 23:33:46.864361 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.864343    2559 scope.go:117] "RemoveContainer" containerID="a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42"
Sep 17 23:33:46.864605 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:33:46.864587    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42\": container with ID starting with a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42 not found: ID does not exist" containerID="a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42"
Sep 17 23:33:46.864651 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.864612    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42"} err="failed to get container status \"a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42\": rpc error: code = NotFound desc = could not find container \"a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42\": container with ID starting with a1ac95760efdf74d162337a54e9be83265f6d6789bc7d58aff7245b36e280d42 not found: ID does not exist"
Sep 17 23:33:46.874975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.874955    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"]
Sep 17 23:33:46.877737 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:46.877690    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/sub-test-model-d90df4-kserve-5cf4b5f78f-5ps6m"]
Sep 17 23:33:47.036615 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.036584    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x" podUID="b9a79ed4-659c-4bc6-a103-29b332eb4edd" containerName="main" containerID="cri-o://5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa" gracePeriod=45
Sep 17 23:33:47.285416 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.285393    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:33:47.454634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454609    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-home\") pod \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") "
Sep 17 23:33:47.454840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454658    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-model-cache\") pod \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") "
Sep 17 23:33:47.454840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454696    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2hhf\" (UniqueName: \"kubernetes.io/projected/b9a79ed4-659c-4bc6-a103-29b332eb4edd-kube-api-access-l2hhf\") pod \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") "
Sep 17 23:33:47.454840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454754    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tls-certs\") pod \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") "
Sep 17 23:33:47.454840 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-dshm\") pod \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") "
Sep 17 23:33:47.455059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454845    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tmp-dir\") pod \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\" (UID: \"b9a79ed4-659c-4bc6-a103-29b332eb4edd\") "
Sep 17 23:33:47.455059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454961    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-home" (OuterVolumeSpecName: "home") pod "b9a79ed4-659c-4bc6-a103-29b332eb4edd" (UID: "b9a79ed4-659c-4bc6-a103-29b332eb4edd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:47.455059 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.454976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9a79ed4-659c-4bc6-a103-29b332eb4edd" (UID: "b9a79ed4-659c-4bc6-a103-29b332eb4edd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:47.455212 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.455170    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:47.455212 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.455190    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:47.455212 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.455188    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9a79ed4-659c-4bc6-a103-29b332eb4edd" (UID: "b9a79ed4-659c-4bc6-a103-29b332eb4edd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:47.457054 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.457026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9a79ed4-659c-4bc6-a103-29b332eb4edd" (UID: "b9a79ed4-659c-4bc6-a103-29b332eb4edd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:47.457380 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.457360    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-dshm" (OuterVolumeSpecName: "dshm") pod "b9a79ed4-659c-4bc6-a103-29b332eb4edd" (UID: "b9a79ed4-659c-4bc6-a103-29b332eb4edd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:47.457440 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.457377    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a79ed4-659c-4bc6-a103-29b332eb4edd-kube-api-access-l2hhf" (OuterVolumeSpecName: "kube-api-access-l2hhf") pod "b9a79ed4-659c-4bc6-a103-29b332eb4edd" (UID: "b9a79ed4-659c-4bc6-a103-29b332eb4edd"). InnerVolumeSpecName "kube-api-access-l2hhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:47.556088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.556062    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2hhf\" (UniqueName: \"kubernetes.io/projected/b9a79ed4-659c-4bc6-a103-29b332eb4edd-kube-api-access-l2hhf\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:47.556088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.556086    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:47.556262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.556095    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:47.556262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.556104    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a79ed4-659c-4bc6-a103-29b332eb4edd-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:47.640969 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.640937    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d5bf60-3451-472f-884d-503c31c32b1a" path="/var/lib/kubelet/pods/86d5bf60-3451-472f-884d-503c31c32b1a/volumes"
Sep 17 23:33:47.861968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.861911    2559 generic.go:356] "Generic (PLEG): container finished" podID="b9a79ed4-659c-4bc6-a103-29b332eb4edd" containerID="5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa" exitCode=0
Sep 17 23:33:47.861968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.861952    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"
Sep 17 23:33:47.861968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.861964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x" event={"ID":"b9a79ed4-659c-4bc6-a103-29b332eb4edd","Type":"ContainerDied","Data":"5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa"}
Sep 17 23:33:47.862148 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.861993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x" event={"ID":"b9a79ed4-659c-4bc6-a103-29b332eb4edd","Type":"ContainerDied","Data":"17e396a734966ac66ff8245d86c1277068d5f1e46f15b725c21fbedd884256d1"}
Sep 17 23:33:47.862148 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.862010    2559 scope.go:117] "RemoveContainer" containerID="5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa"
Sep 17 23:33:47.871193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.871163    2559 scope.go:117] "RemoveContainer" containerID="5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa"
Sep 17 23:33:47.871472 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:33:47.871454    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa\": container with ID starting with 5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa not found: ID does not exist" containerID="5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa"
Sep 17 23:33:47.871574 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.871479    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa"} err="failed to get container status \"5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa\": rpc error: code = NotFound desc = could not find container \"5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa\": container with ID starting with 5d1acab2706be497132c9249321f7f5c15277c62f7266691e9961db96408c7fa not found: ID does not exist"
Sep 17 23:33:47.875815 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.875776    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"]
Sep 17 23:33:47.878254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:47.878231    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d92920/sub-test-model-d92920-kserve-759967d7cf-v4n2x"]
Sep 17 23:33:49.640804 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:49.640767    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a79ed4-659c-4bc6-a103-29b332eb4edd" path="/var/lib/kubelet/pods/b9a79ed4-659c-4bc6-a103-29b332eb4edd/volumes"
Sep 17 23:33:50.283426 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.283385    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"]
Sep 17 23:33:50.288004 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.287976    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d69250-29828130-jczsp"]
Sep 17 23:33:50.317896 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.317867    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"]
Sep 17 23:33:50.318178 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.318127    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm" podUID="0a8247af-b448-4951-bdb2-76f32b55e666" containerName="payload-pre-processing" containerID="cri-o://8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969" gracePeriod=30
Sep 17 23:33:50.581066 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.581040    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:33:50.682563 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.682534    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8247af-b448-4951-bdb2-76f32b55e666-plugins-config-volume\") pod \"0a8247af-b448-4951-bdb2-76f32b55e666\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") "
Sep 17 23:33:50.682925 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.682579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26vjj\" (UniqueName: \"kubernetes.io/projected/0a8247af-b448-4951-bdb2-76f32b55e666-kube-api-access-26vjj\") pod \"0a8247af-b448-4951-bdb2-76f32b55e666\" (UID: \"0a8247af-b448-4951-bdb2-76f32b55e666\") "
Sep 17 23:33:50.682968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.682945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8247af-b448-4951-bdb2-76f32b55e666-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a8247af-b448-4951-bdb2-76f32b55e666" (UID: "0a8247af-b448-4951-bdb2-76f32b55e666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:50.684615 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.684595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8247af-b448-4951-bdb2-76f32b55e666-kube-api-access-26vjj" (OuterVolumeSpecName: "kube-api-access-26vjj") pod "0a8247af-b448-4951-bdb2-76f32b55e666" (UID: "0a8247af-b448-4951-bdb2-76f32b55e666"). InnerVolumeSpecName "kube-api-access-26vjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:50.783288 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.783265    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8247af-b448-4951-bdb2-76f32b55e666-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:50.783288 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.783287    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26vjj\" (UniqueName: \"kubernetes.io/projected/0a8247af-b448-4951-bdb2-76f32b55e666-kube-api-access-26vjj\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:33:50.880399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.880341    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a8247af-b448-4951-bdb2-76f32b55e666" containerID="8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969" exitCode=0
Sep 17 23:33:50.880399 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.880386    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"
Sep 17 23:33:50.880547 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.880430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm" event={"ID":"0a8247af-b448-4951-bdb2-76f32b55e666","Type":"ContainerDied","Data":"8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969"}
Sep 17 23:33:50.880547 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.880476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm" event={"ID":"0a8247af-b448-4951-bdb2-76f32b55e666","Type":"ContainerDied","Data":"a28ead768e2dc87a630d6f237a10f0cd29d3a7ed571df66cabbe12af4cbc9905"}
Sep 17 23:33:50.880547 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.880500    2559 scope.go:117] "RemoveContainer" containerID="8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969"
Sep 17 23:33:50.889950 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.889932    2559 scope.go:117] "RemoveContainer" containerID="8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969"
Sep 17 23:33:50.890222 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:33:50.890203    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969\": container with ID starting with 8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969 not found: ID does not exist" containerID="8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969"
Sep 17 23:33:50.890271 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.890231    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969"} err="failed to get container status \"8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969\": rpc error: code = NotFound desc = could not find container \"8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969\": container with ID starting with 8b199787ffc766629cf78af5d505ba15089d0e81ebbfbf00688a913808b0e969 not found: ID does not exist"
Sep 17 23:33:50.901501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.901479    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"]
Sep 17 23:33:50.905837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:50.905817    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d69250-79d58b9c7d-g9qzm"]
Sep 17 23:33:51.639492 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:51.639461    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a8247af-b448-4951-bdb2-76f32b55e666" path="/var/lib/kubelet/pods/0a8247af-b448-4951-bdb2-76f32b55e666/volumes"
Sep 17 23:33:51.639788 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:51.639774    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26631a94-c281-47c2-a9cd-47cfed5c0f71" path="/var/lib/kubelet/pods/26631a94-c281-47c2-a9cd-47cfed5c0f71/volumes"
Sep 17 23:33:54.928366 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:33:54.928332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:34:00.324222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324185    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"]
Sep 17 23:34:00.324610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324594    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8247af-b448-4951-bdb2-76f32b55e666" containerName="payload-pre-processing"
Sep 17 23:34:00.324610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324610    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8247af-b448-4951-bdb2-76f32b55e666" containerName="payload-pre-processing"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324622    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d5bf60-3451-472f-884d-503c31c32b1a" containerName="main"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324628    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d5bf60-3451-472f-884d-503c31c32b1a" containerName="main"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324636    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ccefb5-ba91-41b6-8b46-8e2aaedbf055" containerName="payload-pre-processing"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324642    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ccefb5-ba91-41b6-8b46-8e2aaedbf055" containerName="payload-pre-processing"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324658    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="664c3663-c9d3-4af1-b1e7-d39b8a5422f3" containerName="payload-processing"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324664    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="664c3663-c9d3-4af1-b1e7-d39b8a5422f3" containerName="payload-processing"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324678    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a79ed4-659c-4bc6-a103-29b332eb4edd" containerName="main"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324683    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a79ed4-659c-4bc6-a103-29b332eb4edd" containerName="main"
Sep 17 23:34:00.324688 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324691    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112edc5d-3266-427a-92fe-bede93d8feed" containerName="payload-pre-processing"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324698    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="112edc5d-3266-427a-92fe-bede93d8feed" containerName="payload-pre-processing"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324779    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a8247af-b448-4951-bdb2-76f32b55e666" containerName="payload-pre-processing"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324791    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d5bf60-3451-472f-884d-503c31c32b1a" containerName="main"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324799    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ccefb5-ba91-41b6-8b46-8e2aaedbf055" containerName="payload-pre-processing"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324805    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="664c3663-c9d3-4af1-b1e7-d39b8a5422f3" containerName="payload-processing"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324811    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="112edc5d-3266-427a-92fe-bede93d8feed" containerName="payload-pre-processing"
Sep 17 23:34:00.325007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.324817    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a79ed4-659c-4bc6-a103-29b332eb4edd" containerName="main"
Sep 17 23:34:00.329860 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.329828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.333000 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.332960    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"openshift-service-ca.crt\""
Sep 17 23:34:00.333554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.333158    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"rate-test-model-d90df4-kserve-self-signed-certs\""
Sep 17 23:34:00.333554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.333248    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"kube-root-ca.crt\""
Sep 17 23:34:00.333554 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.333345    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90df4\"/\"default-dockercfg-6zl8d\""
Sep 17 23:34:00.335638 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.335613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"]
Sep 17 23:34:00.369143 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.369117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9hh\" (UniqueName: \"kubernetes.io/projected/775f112b-412d-4a64-b374-1eebcace5a57-kube-api-access-ft9hh\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.369235 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.369163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-dshm\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.369283 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.369225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-home\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.369320 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.369284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-model-cache\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.369320 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.369314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f112b-412d-4a64-b374-1eebcace5a57-tls-certs\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.369387 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.369338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-tmp-dir\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.470671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.470628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-home\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.470810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.470697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-model-cache\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.470810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.470762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f112b-412d-4a64-b374-1eebcace5a57-tls-certs\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.470810 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.470802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-tmp-dir\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.471029 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.470850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9hh\" (UniqueName: \"kubernetes.io/projected/775f112b-412d-4a64-b374-1eebcace5a57-kube-api-access-ft9hh\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.471029 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.470924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-dshm\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.471155 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.471111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-home\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.471213 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.471179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-model-cache\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.471270 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.471235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-tmp-dir\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.473538 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.473519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-dshm\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.473627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.473603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f112b-412d-4a64-b374-1eebcace5a57-tls-certs\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.478185 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.478162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9hh\" (UniqueName: \"kubernetes.io/projected/775f112b-412d-4a64-b374-1eebcace5a57-kube-api-access-ft9hh\") pod \"rate-test-model-d90df4-kserve-597488685f-b5c84\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") " pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.641208 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.641147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:00.913447 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.913414    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"]
Sep 17 23:34:00.915604 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:34:00.915566    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod775f112b_412d_4a64_b374_1eebcace5a57.slice/crio-deab09620f30ee4014df2a36641af724da07536bb1645c9a8a654818218312bf WatchSource:0}: Error finding container deab09620f30ee4014df2a36641af724da07536bb1645c9a8a654818218312bf: Status 404 returned error can't find the container with id deab09620f30ee4014df2a36641af724da07536bb1645c9a8a654818218312bf
Sep 17 23:34:00.940194 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:00.940166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84" event={"ID":"775f112b-412d-4a64-b374-1eebcace5a57","Type":"ContainerStarted","Data":"deab09620f30ee4014df2a36641af724da07536bb1645c9a8a654818218312bf"}
Sep 17 23:34:01.947314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:01.947271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84" event={"ID":"775f112b-412d-4a64-b374-1eebcace5a57","Type":"ContainerStarted","Data":"1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a"}
Sep 17 23:34:01.963893 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:01.963842    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84" podStartSLOduration=1.963825785 podStartE2EDuration="1.963825785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:01.961674031 +0000 UTC m=+1684.435617014" watchObservedRunningTime="2026-09-17 23:34:01.963825785 +0000 UTC m=+1684.437768768"
Sep 17 23:34:06.042082 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:06.042038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:34:08.027073 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.027029    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"]
Sep 17 23:34:08.029671 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.029636    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fd82a9-29828130-bqbdd"]
Sep 17 23:34:08.049914 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.049886    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"]
Sep 17 23:34:08.050134 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.050085    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94" podUID="b66f453d-1834-4461-84d9-4ca6ed9b2ae3" containerName="payload-processing" containerID="cri-o://783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a" gracePeriod=30
Sep 17 23:34:08.291219 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.291166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:34:08.339641 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.339618    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-plugins-config-volume\") pod \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") "
Sep 17 23:34:08.339772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.339693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2w5m\" (UniqueName: \"kubernetes.io/projected/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-kube-api-access-n2w5m\") pod \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\" (UID: \"b66f453d-1834-4461-84d9-4ca6ed9b2ae3\") "
Sep 17 23:34:08.339981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.339958    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b66f453d-1834-4461-84d9-4ca6ed9b2ae3" (UID: "b66f453d-1834-4461-84d9-4ca6ed9b2ae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:08.341645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.341623    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-kube-api-access-n2w5m" (OuterVolumeSpecName: "kube-api-access-n2w5m") pod "b66f453d-1834-4461-84d9-4ca6ed9b2ae3" (UID: "b66f453d-1834-4461-84d9-4ca6ed9b2ae3"). InnerVolumeSpecName "kube-api-access-n2w5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:08.440856 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.440832    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:34:08.440856 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.440854    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2w5m\" (UniqueName: \"kubernetes.io/projected/b66f453d-1834-4461-84d9-4ca6ed9b2ae3-kube-api-access-n2w5m\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:34:08.991908 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.991874    2559 generic.go:356] "Generic (PLEG): container finished" podID="b66f453d-1834-4461-84d9-4ca6ed9b2ae3" containerID="783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a" exitCode=0
Sep 17 23:34:08.992086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.991924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94" event={"ID":"b66f453d-1834-4461-84d9-4ca6ed9b2ae3","Type":"ContainerDied","Data":"783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a"}
Sep 17 23:34:08.992086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.991967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94" event={"ID":"b66f453d-1834-4461-84d9-4ca6ed9b2ae3","Type":"ContainerDied","Data":"465106e740332fe419b3d02a7ecd419f4d852b91f7fbbeadf0a1c04a19a8d5bb"}
Sep 17 23:34:08.992086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.991990    2559 scope.go:117] "RemoveContainer" containerID="783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a"
Sep 17 23:34:08.992086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:08.991928    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"
Sep 17 23:34:09.002869 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:09.002847    2559 scope.go:117] "RemoveContainer" containerID="783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a"
Sep 17 23:34:09.003172 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:34:09.003146    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a\": container with ID starting with 783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a not found: ID does not exist" containerID="783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a"
Sep 17 23:34:09.003222 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:09.003181    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a"} err="failed to get container status \"783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a\": rpc error: code = NotFound desc = could not find container \"783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a\": container with ID starting with 783ab7ac449c4ab727249bc40d629cc864d445a77adea9c4a9c7b2ed1fff717a not found: ID does not exist"
Sep 17 23:34:09.015765 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:09.015738    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"]
Sep 17 23:34:09.020007 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:09.019980    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fd82a9-695f446579-42d94"]
Sep 17 23:34:09.639964 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:09.639935    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d0c886-813b-4ed8-abc8-0ebd45fb154d" path="/var/lib/kubelet/pods/66d0c886-813b-4ed8-abc8-0ebd45fb154d/volumes"
Sep 17 23:34:09.640313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:09.640284    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66f453d-1834-4461-84d9-4ca6ed9b2ae3" path="/var/lib/kubelet/pods/b66f453d-1834-4461-84d9-4ca6ed9b2ae3/volumes"
Sep 17 23:34:10.641477 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:10.641446    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:10.641873 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:10.641489    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:10.654422 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:10.654389    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:11.015542 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:11.015513    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:34:17.431590 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:17.431512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:34:57.232835 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:34:57.232784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:35:11.224531 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:11.224491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:35:20.925509 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:20.925468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:35:27.153148 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:27.153119    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"]
Sep 17 23:35:32.031420 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:32.031382    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:35:34.486129 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.486096    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"]
Sep 17 23:35:34.486497 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.486487    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66f453d-1834-4461-84d9-4ca6ed9b2ae3" containerName="payload-processing"
Sep 17 23:35:34.486541 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.486499    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66f453d-1834-4461-84d9-4ca6ed9b2ae3" containerName="payload-processing"
Sep 17 23:35:34.486577 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.486566    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66f453d-1834-4461-84d9-4ca6ed9b2ae3" containerName="payload-processing"
Sep 17 23:35:34.489957 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.489938    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.492032 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.492011    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f81ee-dockercfg-ns2mn\""
Sep 17 23:35:34.492139 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.492062    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f81ee\""
Sep 17 23:35:34.499540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.499520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"]
Sep 17 23:35:34.646818 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.646796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c24a3af-8573-4047-a8f3-b32b03d0d875-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.646931 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.646836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2f7\" (UniqueName: \"kubernetes.io/projected/9c24a3af-8573-4047-a8f3-b32b03d0d875-kube-api-access-7q2f7\") pod \"payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.747533 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.747476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c24a3af-8573-4047-a8f3-b32b03d0d875-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.747533 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.747515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2f7\" (UniqueName: \"kubernetes.io/projected/9c24a3af-8573-4047-a8f3-b32b03d0d875-kube-api-access-7q2f7\") pod \"payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.748142 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.748127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c24a3af-8573-4047-a8f3-b32b03d0d875-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.755043 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.755023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2f7\" (UniqueName: \"kubernetes.io/projected/9c24a3af-8573-4047-a8f3-b32b03d0d875-kube-api-access-7q2f7\") pod \"payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:34.801275 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:34.801242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:35.074122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:35.074099    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"]
Sep 17 23:35:35.076114 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:35:35.076086    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c24a3af_8573_4047_a8f3_b32b03d0d875.slice/crio-149e74a994cf8743ec5cabb73ac0e3bd7d4b02615989a50c19bd0445ebbabe03 WatchSource:0}: Error finding container 149e74a994cf8743ec5cabb73ac0e3bd7d4b02615989a50c19bd0445ebbabe03: Status 404 returned error can't find the container with id 149e74a994cf8743ec5cabb73ac0e3bd7d4b02615989a50c19bd0445ebbabe03
Sep 17 23:35:35.467516 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:35.467484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5" event={"ID":"9c24a3af-8573-4047-a8f3-b32b03d0d875","Type":"ContainerStarted","Data":"ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333"}
Sep 17 23:35:35.467704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:35.467553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5" event={"ID":"9c24a3af-8573-4047-a8f3-b32b03d0d875","Type":"ContainerStarted","Data":"149e74a994cf8743ec5cabb73ac0e3bd7d4b02615989a50c19bd0445ebbabe03"}
Sep 17 23:35:35.467704 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:35.467621    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:35.483971 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:35.483934    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5" podStartSLOduration=1.483921885 podStartE2EDuration="1.483921885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:35.481513434 +0000 UTC m=+1777.955456415" watchObservedRunningTime="2026-09-17 23:35:35.483921885 +0000 UTC m=+1777.957864864"
Sep 17 23:35:42.173290 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.173210    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84" podUID="775f112b-412d-4a64-b374-1eebcace5a57" containerName="main" containerID="cri-o://1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a" gracePeriod=45
Sep 17 23:35:42.425273 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.425213    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:35:42.508887 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.508860    2559 generic.go:356] "Generic (PLEG): container finished" podID="775f112b-412d-4a64-b374-1eebcace5a57" containerID="1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a" exitCode=0
Sep 17 23:35:42.509038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.508900    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"
Sep 17 23:35:42.509038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.508943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84" event={"ID":"775f112b-412d-4a64-b374-1eebcace5a57","Type":"ContainerDied","Data":"1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a"}
Sep 17 23:35:42.509038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.508989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84" event={"ID":"775f112b-412d-4a64-b374-1eebcace5a57","Type":"ContainerDied","Data":"deab09620f30ee4014df2a36641af724da07536bb1645c9a8a654818218312bf"}
Sep 17 23:35:42.509038 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.509007    2559 scope.go:117] "RemoveContainer" containerID="1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a"
Sep 17 23:35:42.517168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.517150    2559 scope.go:117] "RemoveContainer" containerID="1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a"
Sep 17 23:35:42.517434 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:35:42.517414    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a\": container with ID starting with 1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a not found: ID does not exist" containerID="1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a"
Sep 17 23:35:42.517496 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.517447    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a"} err="failed to get container status \"1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a\": rpc error: code = NotFound desc = could not find container \"1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a\": container with ID starting with 1b04ac3201d7bad61e8eaa9742fbb2d3ecd216f47e64ae2e4cf20335bc08c57a not found: ID does not exist"
Sep 17 23:35:42.613171 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613147    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-home\") pod \"775f112b-412d-4a64-b374-1eebcace5a57\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") "
Sep 17 23:35:42.613257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-tmp-dir\") pod \"775f112b-412d-4a64-b374-1eebcace5a57\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") "
Sep 17 23:35:42.613257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613231    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-model-cache\") pod \"775f112b-412d-4a64-b374-1eebcace5a57\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") "
Sep 17 23:35:42.613257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613251    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f112b-412d-4a64-b374-1eebcace5a57-tls-certs\") pod \"775f112b-412d-4a64-b374-1eebcace5a57\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") "
Sep 17 23:35:42.613377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft9hh\" (UniqueName: \"kubernetes.io/projected/775f112b-412d-4a64-b374-1eebcace5a57-kube-api-access-ft9hh\") pod \"775f112b-412d-4a64-b374-1eebcace5a57\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") "
Sep 17 23:35:42.613377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613336    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-dshm\") pod \"775f112b-412d-4a64-b374-1eebcace5a57\" (UID: \"775f112b-412d-4a64-b374-1eebcace5a57\") "
Sep 17 23:35:42.613470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613414    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-home" (OuterVolumeSpecName: "home") pod "775f112b-412d-4a64-b374-1eebcace5a57" (UID: "775f112b-412d-4a64-b374-1eebcace5a57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:42.613470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613445    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "775f112b-412d-4a64-b374-1eebcace5a57" (UID: "775f112b-412d-4a64-b374-1eebcace5a57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:42.613470 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613456    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-model-cache" (OuterVolumeSpecName: "model-cache") pod "775f112b-412d-4a64-b374-1eebcace5a57" (UID: "775f112b-412d-4a64-b374-1eebcace5a57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:42.613631 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613614    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:35:42.613631 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613626    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:35:42.613745 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.613638    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:35:42.615398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.615367    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775f112b-412d-4a64-b374-1eebcace5a57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "775f112b-412d-4a64-b374-1eebcace5a57" (UID: "775f112b-412d-4a64-b374-1eebcace5a57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:42.615489 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.615403    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-dshm" (OuterVolumeSpecName: "dshm") pod "775f112b-412d-4a64-b374-1eebcace5a57" (UID: "775f112b-412d-4a64-b374-1eebcace5a57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:42.615489 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.615463    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775f112b-412d-4a64-b374-1eebcace5a57-kube-api-access-ft9hh" (OuterVolumeSpecName: "kube-api-access-ft9hh") pod "775f112b-412d-4a64-b374-1eebcace5a57" (UID: "775f112b-412d-4a64-b374-1eebcace5a57"). InnerVolumeSpecName "kube-api-access-ft9hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:42.714153 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.714132    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f112b-412d-4a64-b374-1eebcace5a57-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:35:42.714153 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.714153    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f112b-412d-4a64-b374-1eebcace5a57-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:35:42.714273 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.714164    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9hh\" (UniqueName: \"kubernetes.io/projected/775f112b-412d-4a64-b374-1eebcace5a57-kube-api-access-ft9hh\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:35:42.829446 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.829422    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"]
Sep 17 23:35:42.832592 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:42.832570    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90df4/rate-test-model-d90df4-kserve-597488685f-b5c84"]
Sep 17 23:35:43.638600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:43.638576    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775f112b-412d-4a64-b374-1eebcace5a57" path="/var/lib/kubelet/pods/775f112b-412d-4a64-b374-1eebcace5a57/volumes"
Sep 17 23:35:44.531044 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:44.531011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:35:46.474262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:46.474238    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:35:59.712962 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:59.712931    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:35:59.718016 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:59.717991    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:35:59.721180 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:59.721161    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:35:59.725749 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:35:59.725713    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:36:04.229377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.229344    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"]
Sep 17 23:36:04.229884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.229775    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775f112b-412d-4a64-b374-1eebcace5a57" containerName="main"
Sep 17 23:36:04.229884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.229790    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="775f112b-412d-4a64-b374-1eebcace5a57" containerName="main"
Sep 17 23:36:04.229884 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.229850    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="775f112b-412d-4a64-b374-1eebcace5a57" containerName="main"
Sep 17 23:36:04.234413 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.234393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.236695 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.236678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-accfb5\""
Sep 17 23:36:04.236805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.236710    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-accfb5-dockercfg-8gvrt\""
Sep 17 23:36:04.240669 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.240645    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"]
Sep 17 23:36:04.268076 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.268051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.268176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.268092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktsb\" (UniqueName: \"kubernetes.io/projected/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-kube-api-access-6ktsb\") pod \"payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.369449 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.369429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktsb\" (UniqueName: \"kubernetes.io/projected/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-kube-api-access-6ktsb\") pod \"payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.369535 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.369515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.370149 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.370133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.376487 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.376466    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktsb\" (UniqueName: \"kubernetes.io/projected/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-kube-api-access-6ktsb\") pod \"payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.546197 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.546147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:04.816823 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:04.816802    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"]
Sep 17 23:36:04.818828 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:36:04.818789    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4e3a37_f35f_4054_a43c_3e0cc04dcb5c.slice/crio-ea95fffa8c65d3f291b7771de849985978d6ea5a1e22f997dddacbb8c75f4271 WatchSource:0}: Error finding container ea95fffa8c65d3f291b7771de849985978d6ea5a1e22f997dddacbb8c75f4271: Status 404 returned error can't find the container with id ea95fffa8c65d3f291b7771de849985978d6ea5a1e22f997dddacbb8c75f4271
Sep 17 23:36:05.622796 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:05.622751    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f" event={"ID":"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c","Type":"ContainerStarted","Data":"99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554"}
Sep 17 23:36:05.623209 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:05.622954    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:05.623209 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:05.622993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f" event={"ID":"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c","Type":"ContainerStarted","Data":"ea95fffa8c65d3f291b7771de849985978d6ea5a1e22f997dddacbb8c75f4271"}
Sep 17 23:36:05.640206 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:05.640160    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f" podStartSLOduration=1.640149568 podStartE2EDuration="1.640149568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:05.6382378 +0000 UTC m=+1808.112180784" watchObservedRunningTime="2026-09-17 23:36:05.640149568 +0000 UTC m=+1808.114092606"
Sep 17 23:36:09.426349 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:09.426318    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:36:16.629253 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:16.629222    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:36:31.509068 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:31.509038    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f_3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c/payload-processing/0.log"
Sep 17 23:36:39.623172 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.623141    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"]
Sep 17 23:36:39.626692 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.626667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.630096 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.630070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8f81ee\"/\"ipp-route-8f81ee-kserve-self-signed-certs\""
Sep 17 23:36:39.630254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.630105    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8f81ee\"/\"openshift-service-ca.crt\""
Sep 17 23:36:39.630254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.630115    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8f81ee\"/\"kube-root-ca.crt\""
Sep 17 23:36:39.630254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.630160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8f81ee\"/\"default-dockercfg-dlt97\""
Sep 17 23:36:39.641689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.641665    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"]
Sep 17 23:36:39.742846 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.742821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhd5\" (UniqueName: \"kubernetes.io/projected/1fc78fef-4a8e-4deb-880a-526ee5b8c575-kube-api-access-7dhd5\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.743022 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.743001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-home\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.743086 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.743056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tmp-dir\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.743212 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.743192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tls-certs\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.743286 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.743234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-model-cache\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.743680 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.743636    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-dshm\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844498 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-home\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tmp-dir\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tls-certs\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-model-cache\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-dshm\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844830 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhd5\" (UniqueName: \"kubernetes.io/projected/1fc78fef-4a8e-4deb-880a-526ee5b8c575-kube-api-access-7dhd5\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844890 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-home\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844947 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844905    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tmp-dir\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.844947 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.844934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-model-cache\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.846834 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.846809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-dshm\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.847097 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.847081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tls-certs\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.851805 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.851783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhd5\" (UniqueName: \"kubernetes.io/projected/1fc78fef-4a8e-4deb-880a-526ee5b8c575-kube-api-access-7dhd5\") pod \"ipp-route-8f81ee-kserve-86f6b669f9-72lcp\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") " pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:39.938236 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:39.938186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:40.208248 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:40.208216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"]
Sep 17 23:36:40.211002 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:36:40.210974    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fc78fef_4a8e_4deb_880a_526ee5b8c575.slice/crio-7503bd35bcd0f8a12b87c9790920d0c38ff354ae17e604a8f136d35eba87a21f WatchSource:0}: Error finding container 7503bd35bcd0f8a12b87c9790920d0c38ff354ae17e604a8f136d35eba87a21f: Status 404 returned error can't find the container with id 7503bd35bcd0f8a12b87c9790920d0c38ff354ae17e604a8f136d35eba87a21f
Sep 17 23:36:40.799896 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:40.799866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp" event={"ID":"1fc78fef-4a8e-4deb-880a-526ee5b8c575","Type":"ContainerStarted","Data":"88e51a59426b32869783bfde3e4a4e8a1053deaa000c2aee5144fe4ab2b7638d"}
Sep 17 23:36:40.800276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:40.799901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp" event={"ID":"1fc78fef-4a8e-4deb-880a-526ee5b8c575","Type":"ContainerStarted","Data":"7503bd35bcd0f8a12b87c9790920d0c38ff354ae17e604a8f136d35eba87a21f"}
Sep 17 23:36:40.815386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:40.815342    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp" podStartSLOduration=1.815324441 podStartE2EDuration="1.815324441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:40.814398571 +0000 UTC m=+1843.288341553" watchObservedRunningTime="2026-09-17 23:36:40.815324441 +0000 UTC m=+1843.289267426"
Sep 17 23:36:43.927033 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:43.926998    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:36:49.938560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:49.938526    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:49.938560 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:49.938569    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:49.950412 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:49.950388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:50.862279 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:50.862251    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:36:56.632673 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:56.632641    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:36:59.933484 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:59.933454    2559 scope.go:117] "RemoveContainer" containerID="9e4a8c98454cf0d209e89039b98de75655aaf748fd0994f1bb7c70a9172601a2"
Sep 17 23:36:59.941836 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:36:59.941811    2559 scope.go:117] "RemoveContainer" containerID="5990eec413c53bfd2bebf64cc791301560727b04662b816f8fbb06cf7d08332d"
Sep 17 23:37:02.461271 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:02.461241    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5_9c24a3af-8573-4047-a8f3-b32b03d0d875/payload-processing/0.log"
Sep 17 23:37:02.712880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:02.712800    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f_3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c/payload-processing/0.log"
Sep 17 23:37:10.398665 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.398623    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"]
Sep 17 23:37:10.403902 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.403879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.406190 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.406163    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e485c4-dockercfg-rhd4h\""
Sep 17 23:37:10.406335 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.406196    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e485c4\""
Sep 17 23:37:10.412734 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.412695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"]
Sep 17 23:37:10.425063 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.425037    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"]
Sep 17 23:37:10.428540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.428522    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.439454 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.439426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"]
Sep 17 23:37:10.507613 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.507574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphf7\" (UniqueName: \"kubernetes.io/projected/f5292eec-5261-40a5-bce9-2afaf5c4660c-kube-api-access-gphf7\") pod \"payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.507811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.507632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.507811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.507659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5292eec-5261-40a5-bce9-2afaf5c4660c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.507811 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.507770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8c6\" (UniqueName: \"kubernetes.io/projected/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-kube-api-access-hw8c6\") pod \"payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.608378 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.608344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphf7\" (UniqueName: \"kubernetes.io/projected/f5292eec-5261-40a5-bce9-2afaf5c4660c-kube-api-access-gphf7\") pod \"payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.608530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.608384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.608530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.608403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5292eec-5261-40a5-bce9-2afaf5c4660c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.608530 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.608441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8c6\" (UniqueName: \"kubernetes.io/projected/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-kube-api-access-hw8c6\") pod \"payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.609155 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.609137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.609210 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.609162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5292eec-5261-40a5-bce9-2afaf5c4660c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.616405 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.616382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphf7\" (UniqueName: \"kubernetes.io/projected/f5292eec-5261-40a5-bce9-2afaf5c4660c-kube-api-access-gphf7\") pod \"payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.616405 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.616393    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8c6\" (UniqueName: \"kubernetes.io/projected/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-kube-api-access-hw8c6\") pod \"payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.715540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.715510    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:10.738521 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.738494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:10.988601 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:10.988534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"]
Sep 17 23:37:10.991513 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:37:10.991480    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5292eec_5261_40a5_bce9_2afaf5c4660c.slice/crio-108b94ce21279d092e726298cc39d69441c2860420ada07c5480582e58218019 WatchSource:0}: Error finding container 108b94ce21279d092e726298cc39d69441c2860420ada07c5480582e58218019: Status 404 returned error can't find the container with id 108b94ce21279d092e726298cc39d69441c2860420ada07c5480582e58218019
Sep 17 23:37:11.022157 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.022134    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"]
Sep 17 23:37:11.024036 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:37:11.024011    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098d747f_2aa1_4ad9_8c51_50a40ae09a1e.slice/crio-144339967dd7044b3cda9bdbc8e79c233a4ae2b12d1d0b22f04fd2102128ecca WatchSource:0}: Error finding container 144339967dd7044b3cda9bdbc8e79c233a4ae2b12d1d0b22f04fd2102128ecca: Status 404 returned error can't find the container with id 144339967dd7044b3cda9bdbc8e79c233a4ae2b12d1d0b22f04fd2102128ecca
Sep 17 23:37:11.964093 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.964052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d" event={"ID":"f5292eec-5261-40a5-bce9-2afaf5c4660c","Type":"ContainerStarted","Data":"aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3"}
Sep 17 23:37:11.964498 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.964106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d" event={"ID":"f5292eec-5261-40a5-bce9-2afaf5c4660c","Type":"ContainerStarted","Data":"108b94ce21279d092e726298cc39d69441c2860420ada07c5480582e58218019"}
Sep 17 23:37:11.964498 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.964132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:11.965740 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.965701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4" event={"ID":"098d747f-2aa1-4ad9-8c51-50a40ae09a1e","Type":"ContainerStarted","Data":"6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8"}
Sep 17 23:37:11.965852 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.965753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4" event={"ID":"098d747f-2aa1-4ad9-8c51-50a40ae09a1e","Type":"ContainerStarted","Data":"144339967dd7044b3cda9bdbc8e79c233a4ae2b12d1d0b22f04fd2102128ecca"}
Sep 17 23:37:11.965852 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.965819    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:11.979460 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.979419    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d" podStartSLOduration=1.979404475 podStartE2EDuration="1.979404475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:11.978449591 +0000 UTC m=+1874.452392574" watchObservedRunningTime="2026-09-17 23:37:11.979404475 +0000 UTC m=+1874.453347459"
Sep 17 23:37:11.994918 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:11.994872    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4" podStartSLOduration=1.9948578110000001 podStartE2EDuration="1.994857811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:11.993768434 +0000 UTC m=+1874.467711416" watchObservedRunningTime="2026-09-17 23:37:11.994857811 +0000 UTC m=+1874.468800794"
Sep 17 23:37:16.628377 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:16.628343    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:37:22.972574 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:22.972542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:22.973031 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:22.972614    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:48.517136 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.517104    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"]
Sep 17 23:37:48.517687 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.517304    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4" podUID="098d747f-2aa1-4ad9-8c51-50a40ae09a1e" containerName="payload-processing" containerID="cri-o://6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8" gracePeriod=30
Sep 17 23:37:48.523151 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.523125    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"]
Sep 17 23:37:48.523345 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.523304    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d" podUID="f5292eec-5261-40a5-bce9-2afaf5c4660c" containerName="payload-pre-processing" containerID="cri-o://aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3" gracePeriod=30
Sep 17 23:37:48.786150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.786126    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:48.808205 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.808183    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:48.860767 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.860740    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gphf7\" (UniqueName: \"kubernetes.io/projected/f5292eec-5261-40a5-bce9-2afaf5c4660c-kube-api-access-gphf7\") pod \"f5292eec-5261-40a5-bce9-2afaf5c4660c\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") "
Sep 17 23:37:48.860880 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.860814    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5292eec-5261-40a5-bce9-2afaf5c4660c-plugins-config-volume\") pod \"f5292eec-5261-40a5-bce9-2afaf5c4660c\" (UID: \"f5292eec-5261-40a5-bce9-2afaf5c4660c\") "
Sep 17 23:37:48.860928 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.860908    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-plugins-config-volume\") pod \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") "
Sep 17 23:37:48.860991 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.860977    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw8c6\" (UniqueName: \"kubernetes.io/projected/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-kube-api-access-hw8c6\") pod \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\" (UID: \"098d747f-2aa1-4ad9-8c51-50a40ae09a1e\") "
Sep 17 23:37:48.861219 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.861191    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5292eec-5261-40a5-bce9-2afaf5c4660c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5292eec-5261-40a5-bce9-2afaf5c4660c" (UID: "f5292eec-5261-40a5-bce9-2afaf5c4660c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:48.861315 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.861238    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "098d747f-2aa1-4ad9-8c51-50a40ae09a1e" (UID: "098d747f-2aa1-4ad9-8c51-50a40ae09a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:48.862848 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.862830    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5292eec-5261-40a5-bce9-2afaf5c4660c-kube-api-access-gphf7" (OuterVolumeSpecName: "kube-api-access-gphf7") pod "f5292eec-5261-40a5-bce9-2afaf5c4660c" (UID: "f5292eec-5261-40a5-bce9-2afaf5c4660c"). InnerVolumeSpecName "kube-api-access-gphf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:48.862921 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.862888    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-kube-api-access-hw8c6" (OuterVolumeSpecName: "kube-api-access-hw8c6") pod "098d747f-2aa1-4ad9-8c51-50a40ae09a1e" (UID: "098d747f-2aa1-4ad9-8c51-50a40ae09a1e"). InnerVolumeSpecName "kube-api-access-hw8c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:48.962312 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.962291    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:37:48.962312 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.962312    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw8c6\" (UniqueName: \"kubernetes.io/projected/098d747f-2aa1-4ad9-8c51-50a40ae09a1e-kube-api-access-hw8c6\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:37:48.962444 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.962322    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gphf7\" (UniqueName: \"kubernetes.io/projected/f5292eec-5261-40a5-bce9-2afaf5c4660c-kube-api-access-gphf7\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:37:48.962444 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:48.962330    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5292eec-5261-40a5-bce9-2afaf5c4660c-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:37:49.155878 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.155820    2559 generic.go:356] "Generic (PLEG): container finished" podID="098d747f-2aa1-4ad9-8c51-50a40ae09a1e" containerID="6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8" exitCode=0
Sep 17 23:37:49.155878 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.155868    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"
Sep 17 23:37:49.156032 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.155912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4" event={"ID":"098d747f-2aa1-4ad9-8c51-50a40ae09a1e","Type":"ContainerDied","Data":"6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8"}
Sep 17 23:37:49.156032 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.155962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4" event={"ID":"098d747f-2aa1-4ad9-8c51-50a40ae09a1e","Type":"ContainerDied","Data":"144339967dd7044b3cda9bdbc8e79c233a4ae2b12d1d0b22f04fd2102128ecca"}
Sep 17 23:37:49.156032 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.155989    2559 scope.go:117] "RemoveContainer" containerID="6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8"
Sep 17 23:37:49.157390 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.157364    2559 generic.go:356] "Generic (PLEG): container finished" podID="f5292eec-5261-40a5-bce9-2afaf5c4660c" containerID="aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3" exitCode=0
Sep 17 23:37:49.157501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.157408    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"
Sep 17 23:37:49.157501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.157434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d" event={"ID":"f5292eec-5261-40a5-bce9-2afaf5c4660c","Type":"ContainerDied","Data":"aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3"}
Sep 17 23:37:49.157501 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.157477    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d" event={"ID":"f5292eec-5261-40a5-bce9-2afaf5c4660c","Type":"ContainerDied","Data":"108b94ce21279d092e726298cc39d69441c2860420ada07c5480582e58218019"}
Sep 17 23:37:49.166183 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.166166    2559 scope.go:117] "RemoveContainer" containerID="6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8"
Sep 17 23:37:49.166428 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:37:49.166402    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8\": container with ID starting with 6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8 not found: ID does not exist" containerID="6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8"
Sep 17 23:37:49.166493 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.166435    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8"} err="failed to get container status \"6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8\": rpc error: code = NotFound desc = could not find container \"6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8\": container with ID starting with 6c08a57f42cf18dcd1f3afee20a615ad93823660d19628f55457017685fe03a8 not found: ID does not exist"
Sep 17 23:37:49.166493 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.166451    2559 scope.go:117] "RemoveContainer" containerID="aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3"
Sep 17 23:37:49.174400 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.174384    2559 scope.go:117] "RemoveContainer" containerID="aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3"
Sep 17 23:37:49.174657 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:37:49.174638    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3\": container with ID starting with aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3 not found: ID does not exist" containerID="aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3"
Sep 17 23:37:49.174730 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.174662    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3"} err="failed to get container status \"aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3\": rpc error: code = NotFound desc = could not find container \"aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3\": container with ID starting with aee83aa6535ec57f624631fede4aac5dd304e7994330eec88e888a30e05d48d3 not found: ID does not exist"
Sep 17 23:37:49.177983 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.177960    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"]
Sep 17 23:37:49.183228 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.183208    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e485c4-6f765dbfc-w6tl4"]
Sep 17 23:37:49.195430 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.195411    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"]
Sep 17 23:37:49.207814 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.207794    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e485c4-6ffc5c46c4-lpz7d"]
Sep 17 23:37:49.638950 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.638928    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098d747f-2aa1-4ad9-8c51-50a40ae09a1e" path="/var/lib/kubelet/pods/098d747f-2aa1-4ad9-8c51-50a40ae09a1e/volumes"
Sep 17 23:37:49.639270 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:49.639257    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5292eec-5261-40a5-bce9-2afaf5c4660c" path="/var/lib/kubelet/pods/f5292eec-5261-40a5-bce9-2afaf5c4660c/volumes"
Sep 17 23:37:53.328435 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:37:53.328403    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:38:12.224577 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:12.224542    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:38:14.733348 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:14.733316    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"]
Sep 17 23:38:14.733795 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:14.733516    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5" podUID="9c24a3af-8573-4047-a8f3-b32b03d0d875" containerName="payload-processing" containerID="cri-o://ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333" gracePeriod=30
Sep 17 23:38:14.988389 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:14.988326    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:38:15.098777 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.098741    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c24a3af-8573-4047-a8f3-b32b03d0d875-plugins-config-volume\") pod \"9c24a3af-8573-4047-a8f3-b32b03d0d875\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") "
Sep 17 23:38:15.098936 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.098813    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q2f7\" (UniqueName: \"kubernetes.io/projected/9c24a3af-8573-4047-a8f3-b32b03d0d875-kube-api-access-7q2f7\") pod \"9c24a3af-8573-4047-a8f3-b32b03d0d875\" (UID: \"9c24a3af-8573-4047-a8f3-b32b03d0d875\") "
Sep 17 23:38:15.099160 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.099132    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c24a3af-8573-4047-a8f3-b32b03d0d875-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c24a3af-8573-4047-a8f3-b32b03d0d875" (UID: "9c24a3af-8573-4047-a8f3-b32b03d0d875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:15.100939 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.100916    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c24a3af-8573-4047-a8f3-b32b03d0d875-kube-api-access-7q2f7" (OuterVolumeSpecName: "kube-api-access-7q2f7") pod "9c24a3af-8573-4047-a8f3-b32b03d0d875" (UID: "9c24a3af-8573-4047-a8f3-b32b03d0d875"). InnerVolumeSpecName "kube-api-access-7q2f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:15.200237 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.200204    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c24a3af-8573-4047-a8f3-b32b03d0d875-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:15.200237 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.200229    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2f7\" (UniqueName: \"kubernetes.io/projected/9c24a3af-8573-4047-a8f3-b32b03d0d875-kube-api-access-7q2f7\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:15.289259 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.289170    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c24a3af-8573-4047-a8f3-b32b03d0d875" containerID="ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333" exitCode=0
Sep 17 23:38:15.289259 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.289214    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"
Sep 17 23:38:15.289259 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.289217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5" event={"ID":"9c24a3af-8573-4047-a8f3-b32b03d0d875","Type":"ContainerDied","Data":"ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333"}
Sep 17 23:38:15.289471 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.289269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5" event={"ID":"9c24a3af-8573-4047-a8f3-b32b03d0d875","Type":"ContainerDied","Data":"149e74a994cf8743ec5cabb73ac0e3bd7d4b02615989a50c19bd0445ebbabe03"}
Sep 17 23:38:15.289471 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.289299    2559 scope.go:117] "RemoveContainer" containerID="ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333"
Sep 17 23:38:15.297938 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.297921    2559 scope.go:117] "RemoveContainer" containerID="ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333"
Sep 17 23:38:15.298210 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:38:15.298190    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333\": container with ID starting with ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333 not found: ID does not exist" containerID="ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333"
Sep 17 23:38:15.298279 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.298224    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333"} err="failed to get container status \"ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333\": rpc error: code = NotFound desc = could not find container \"ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333\": container with ID starting with ec2997c2e710670e392075e5cb82bddaa52e175b34e4c3d9605905f82da3c333 not found: ID does not exist"
Sep 17 23:38:15.309462 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.309439    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"]
Sep 17 23:38:15.313413 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.313391    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f81ee-d5996f667-5llp5"]
Sep 17 23:38:15.648118 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:15.648026    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c24a3af-8573-4047-a8f3-b32b03d0d875" path="/var/lib/kubelet/pods/9c24a3af-8573-4047-a8f3-b32b03d0d875/volumes"
Sep 17 23:38:20.271242 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:20.271202    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"]
Sep 17 23:38:25.325506 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:25.325463    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:38:35.290329 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.290283    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp" podUID="1fc78fef-4a8e-4deb-880a-526ee5b8c575" containerName="main" containerID="cri-o://88e51a59426b32869783bfde3e4a4e8a1053deaa000c2aee5144fe4ab2b7638d" gracePeriod=45
Sep 17 23:38:35.407245 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.407200    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fc78fef-4a8e-4deb-880a-526ee5b8c575" containerID="88e51a59426b32869783bfde3e4a4e8a1053deaa000c2aee5144fe4ab2b7638d" exitCode=0
Sep 17 23:38:35.407305 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.407267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp" event={"ID":"1fc78fef-4a8e-4deb-880a-526ee5b8c575","Type":"ContainerDied","Data":"88e51a59426b32869783bfde3e4a4e8a1053deaa000c2aee5144fe4ab2b7638d"}
Sep 17 23:38:35.542944 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.542875    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:38:35.590627 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.590583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tls-certs\") pod \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") "
Sep 17 23:38:35.590834 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.590639    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-dshm\") pod \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") "
Sep 17 23:38:35.590834 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.590688    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tmp-dir\") pod \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") "
Sep 17 23:38:35.591047 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.590739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-model-cache\") pod \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") "
Sep 17 23:38:35.591150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-home\") pod \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") "
Sep 17 23:38:35.591278 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591165    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dhd5\" (UniqueName: \"kubernetes.io/projected/1fc78fef-4a8e-4deb-880a-526ee5b8c575-kube-api-access-7dhd5\") pod \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\" (UID: \"1fc78fef-4a8e-4deb-880a-526ee5b8c575\") "
Sep 17 23:38:35.591382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.590998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fc78fef-4a8e-4deb-880a-526ee5b8c575" (UID: "1fc78fef-4a8e-4deb-880a-526ee5b8c575"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:35.591382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591014    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fc78fef-4a8e-4deb-880a-526ee5b8c575" (UID: "1fc78fef-4a8e-4deb-880a-526ee5b8c575"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:35.591382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591348    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-home" (OuterVolumeSpecName: "home") pod "1fc78fef-4a8e-4deb-880a-526ee5b8c575" (UID: "1fc78fef-4a8e-4deb-880a-526ee5b8c575"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:35.591598 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591549    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:35.591598 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591572    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:35.591598 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.591584    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:35.593212 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.593185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-dshm" (OuterVolumeSpecName: "dshm") pod "1fc78fef-4a8e-4deb-880a-526ee5b8c575" (UID: "1fc78fef-4a8e-4deb-880a-526ee5b8c575"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:35.593314 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.593252    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fc78fef-4a8e-4deb-880a-526ee5b8c575" (UID: "1fc78fef-4a8e-4deb-880a-526ee5b8c575"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:38:35.593739 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.593702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc78fef-4a8e-4deb-880a-526ee5b8c575-kube-api-access-7dhd5" (OuterVolumeSpecName: "kube-api-access-7dhd5") pod "1fc78fef-4a8e-4deb-880a-526ee5b8c575" (UID: "1fc78fef-4a8e-4deb-880a-526ee5b8c575"). InnerVolumeSpecName "kube-api-access-7dhd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:35.692919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.692876    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc78fef-4a8e-4deb-880a-526ee5b8c575-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:35.692919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.692911    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc78fef-4a8e-4deb-880a-526ee5b8c575-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:35.692919 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:35.692921    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhd5\" (UniqueName: \"kubernetes.io/projected/1fc78fef-4a8e-4deb-880a-526ee5b8c575-kube-api-access-7dhd5\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:38:36.414026 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:36.413993    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"
Sep 17 23:38:36.414450 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:36.413993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp" event={"ID":"1fc78fef-4a8e-4deb-880a-526ee5b8c575","Type":"ContainerDied","Data":"7503bd35bcd0f8a12b87c9790920d0c38ff354ae17e604a8f136d35eba87a21f"}
Sep 17 23:38:36.414450 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:36.414123    2559 scope.go:117] "RemoveContainer" containerID="88e51a59426b32869783bfde3e4a4e8a1053deaa000c2aee5144fe4ab2b7638d"
Sep 17 23:38:36.430182 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:36.430147    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"]
Sep 17 23:38:36.433906 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:36.433879    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8f81ee/ipp-route-8f81ee-kserve-86f6b669f9-72lcp"]
Sep 17 23:38:37.639461 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:38:37.639427    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc78fef-4a8e-4deb-880a-526ee5b8c575" path="/var/lib/kubelet/pods/1fc78fef-4a8e-4deb-880a-526ee5b8c575/volumes"
Sep 17 23:39:14.236862 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.236823    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"]
Sep 17 23:39:14.237379 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.237030    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f" podUID="3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" containerName="payload-processing" containerID="cri-o://99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554" gracePeriod=30
Sep 17 23:39:14.492020 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.491964    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:39:14.549407 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.549378    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-plugins-config-volume\") pod \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") "
Sep 17 23:39:14.549562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.549426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ktsb\" (UniqueName: \"kubernetes.io/projected/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-kube-api-access-6ktsb\") pod \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\" (UID: \"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c\") "
Sep 17 23:39:14.549825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.549803    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" (UID: "3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:14.551570 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.551550    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-kube-api-access-6ktsb" (OuterVolumeSpecName: "kube-api-access-6ktsb") pod "3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" (UID: "3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c"). InnerVolumeSpecName "kube-api-access-6ktsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:14.615684 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.615651    2559 generic.go:356] "Generic (PLEG): container finished" podID="3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" containerID="99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554" exitCode=0
Sep 17 23:39:14.615843 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.615695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"
Sep 17 23:39:14.615843 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.615686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f" event={"ID":"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c","Type":"ContainerDied","Data":"99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554"}
Sep 17 23:39:14.615843 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.615817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f" event={"ID":"3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c","Type":"ContainerDied","Data":"ea95fffa8c65d3f291b7771de849985978d6ea5a1e22f997dddacbb8c75f4271"}
Sep 17 23:39:14.615843 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.615838    2559 scope.go:117] "RemoveContainer" containerID="99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554"
Sep 17 23:39:14.624669 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.624654    2559 scope.go:117] "RemoveContainer" containerID="99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554"
Sep 17 23:39:14.624958 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:39:14.624931    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554\": container with ID starting with 99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554 not found: ID does not exist" containerID="99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554"
Sep 17 23:39:14.625021 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.624960    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554"} err="failed to get container status \"99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554\": rpc error: code = NotFound desc = could not find container \"99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554\": container with ID starting with 99d79121bc1f2c13b97ad51175e09674488f12644eedab50a836e10294294554 not found: ID does not exist"
Sep 17 23:39:14.638206 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.638183    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"]
Sep 17 23:39:14.642357 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.642337    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-accfb5-ccbdb46c-4kb8f"]
Sep 17 23:39:14.650905 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.650872    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:14.650905 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:14.650905    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ktsb\" (UniqueName: \"kubernetes.io/projected/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c-kube-api-access-6ktsb\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:15.639146 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:15.639117    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" path="/var/lib/kubelet/pods/3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c/volumes"
Sep 17 23:39:17.530986 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:17.530948    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:39:31.642776 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:31.642744    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"]
Sep 17 23:39:31.643276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:31.642815    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"]
Sep 17 23:39:31.643276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:31.643009    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8" podUID="7aa04723-12bd-4525-b26f-f6e1d1841ccb" containerName="payload-pre-processing" containerID="cri-o://a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc" gracePeriod=30
Sep 17 23:39:31.643276 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:31.643099    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp" podUID="b87e9df5-6d4e-40c4-8bc6-f513f29c001a" containerName="payload-processing" containerID="cri-o://fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713" gracePeriod=30
Sep 17 23:39:31.899779 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:31.899753    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:39:31.913394 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:31.913372    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:39:32.008117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.008089    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rm9v\" (UniqueName: \"kubernetes.io/projected/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-kube-api-access-6rm9v\") pod \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") "
Sep 17 23:39:32.008262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.008144    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8xqm\" (UniqueName: \"kubernetes.io/projected/7aa04723-12bd-4525-b26f-f6e1d1841ccb-kube-api-access-r8xqm\") pod \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") "
Sep 17 23:39:32.008262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.008217    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa04723-12bd-4525-b26f-f6e1d1841ccb-plugins-config-volume\") pod \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\" (UID: \"7aa04723-12bd-4525-b26f-f6e1d1841ccb\") "
Sep 17 23:39:32.008262 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.008246    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-plugins-config-volume\") pod \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\" (UID: \"b87e9df5-6d4e-40c4-8bc6-f513f29c001a\") "
Sep 17 23:39:32.008574 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.008551    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa04723-12bd-4525-b26f-f6e1d1841ccb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7aa04723-12bd-4525-b26f-f6e1d1841ccb" (UID: "7aa04723-12bd-4525-b26f-f6e1d1841ccb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:32.008650 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.008606    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b87e9df5-6d4e-40c4-8bc6-f513f29c001a" (UID: "b87e9df5-6d4e-40c4-8bc6-f513f29c001a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:32.010232 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.010213    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-kube-api-access-6rm9v" (OuterVolumeSpecName: "kube-api-access-6rm9v") pod "b87e9df5-6d4e-40c4-8bc6-f513f29c001a" (UID: "b87e9df5-6d4e-40c4-8bc6-f513f29c001a"). InnerVolumeSpecName "kube-api-access-6rm9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:32.010293 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.010247    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa04723-12bd-4525-b26f-f6e1d1841ccb-kube-api-access-r8xqm" (OuterVolumeSpecName: "kube-api-access-r8xqm") pod "7aa04723-12bd-4525-b26f-f6e1d1841ccb" (UID: "7aa04723-12bd-4525-b26f-f6e1d1841ccb"). InnerVolumeSpecName "kube-api-access-r8xqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:32.109428 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.109405    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa04723-12bd-4525-b26f-f6e1d1841ccb-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:32.109428 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.109425    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:32.109564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.109433    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rm9v\" (UniqueName: \"kubernetes.io/projected/b87e9df5-6d4e-40c4-8bc6-f513f29c001a-kube-api-access-6rm9v\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:32.109564 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.109441    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xqm\" (UniqueName: \"kubernetes.io/projected/7aa04723-12bd-4525-b26f-f6e1d1841ccb-kube-api-access-r8xqm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:32.700633 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.700602    2559 generic.go:356] "Generic (PLEG): container finished" podID="7aa04723-12bd-4525-b26f-f6e1d1841ccb" containerID="a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc" exitCode=0
Sep 17 23:39:32.701040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.700639    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"
Sep 17 23:39:32.701040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.700685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8" event={"ID":"7aa04723-12bd-4525-b26f-f6e1d1841ccb","Type":"ContainerDied","Data":"a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc"}
Sep 17 23:39:32.701040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.700746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8" event={"ID":"7aa04723-12bd-4525-b26f-f6e1d1841ccb","Type":"ContainerDied","Data":"2e3422740524beeb8e12da4f66e291e215062666233c17c33128a5a93ca9737b"}
Sep 17 23:39:32.701040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.700771    2559 scope.go:117] "RemoveContainer" containerID="a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc"
Sep 17 23:39:32.702107 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.702086    2559 generic.go:356] "Generic (PLEG): container finished" podID="b87e9df5-6d4e-40c4-8bc6-f513f29c001a" containerID="fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713" exitCode=0
Sep 17 23:39:32.702176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.702128    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"
Sep 17 23:39:32.702176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.702162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp" event={"ID":"b87e9df5-6d4e-40c4-8bc6-f513f29c001a","Type":"ContainerDied","Data":"fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713"}
Sep 17 23:39:32.702253 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.702200    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp" event={"ID":"b87e9df5-6d4e-40c4-8bc6-f513f29c001a","Type":"ContainerDied","Data":"190b4443739a84118a826cfc84df8259de23a5dff07de34f99b9a84de11bde3a"}
Sep 17 23:39:32.710257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.710223    2559 scope.go:117] "RemoveContainer" containerID="a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc"
Sep 17 23:39:32.710538 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:39:32.710508    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc\": container with ID starting with a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc not found: ID does not exist" containerID="a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc"
Sep 17 23:39:32.710632 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.710547    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc"} err="failed to get container status \"a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc\": rpc error: code = NotFound desc = could not find container \"a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc\": container with ID starting with a5bb2b060e5fe38fedee5a2f6eb527f560a9f601ac4f359d20e2bbf49993a8cc not found: ID does not exist"
Sep 17 23:39:32.710632 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.710571    2559 scope.go:117] "RemoveContainer" containerID="fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713"
Sep 17 23:39:32.718928 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.718911    2559 scope.go:117] "RemoveContainer" containerID="fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713"
Sep 17 23:39:32.719162 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:39:32.719144    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713\": container with ID starting with fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713 not found: ID does not exist" containerID="fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713"
Sep 17 23:39:32.719223 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.719172    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713"} err="failed to get container status \"fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713\": rpc error: code = NotFound desc = could not find container \"fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713\": container with ID starting with fbd3d56bdc6fe37bb198b6268830377fda2360d460261a6d4fd5b3bbd5b93713 not found: ID does not exist"
Sep 17 23:39:32.725469 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.725449    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"]
Sep 17 23:39:32.728921 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.728901    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90df4-75745866c7-r9nfp"]
Sep 17 23:39:32.741344 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.741322    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"]
Sep 17 23:39:32.744877 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:32.744856    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90df4-544d54dd86-7kgr8"]
Sep 17 23:39:33.638546 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:33.638513    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa04723-12bd-4525-b26f-f6e1d1841ccb" path="/var/lib/kubelet/pods/7aa04723-12bd-4525-b26f-f6e1d1841ccb/volumes"
Sep 17 23:39:33.638891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:33.638878    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87e9df5-6d4e-40c4-8bc6-f513f29c001a" path="/var/lib/kubelet/pods/b87e9df5-6d4e-40c4-8bc6-f513f29c001a/volumes"
Sep 17 23:39:34.427133 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:34.427102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:39:49.231024 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.230942    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"]
Sep 17 23:39:49.233121 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.233097    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d92920-29828130-t6n48"]
Sep 17 23:39:49.256539 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.256514    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"]
Sep 17 23:39:49.256790 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.256757    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s" podUID="5c7d3d6c-7876-4c62-91d4-d81c83d40969" containerName="payload-processing" containerID="cri-o://8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729" gracePeriod=30
Sep 17 23:39:49.263981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.263700    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"]
Sep 17 23:39:49.263981 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.263922    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg" podUID="ccc91222-888f-47a5-aa12-af6164fe2738" containerName="payload-pre-processing" containerID="cri-o://63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5" gracePeriod=30
Sep 17 23:39:49.526254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.526229    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:39:49.528971 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.528950    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:39:49.638628 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.638598    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4625ab98-a6fc-4033-b18c-77ba0611ef46" path="/var/lib/kubelet/pods/4625ab98-a6fc-4033-b18c-77ba0611ef46/volumes"
Sep 17 23:39:49.650669 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.650649    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b72sr\" (UniqueName: \"kubernetes.io/projected/5c7d3d6c-7876-4c62-91d4-d81c83d40969-kube-api-access-b72sr\") pod \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") "
Sep 17 23:39:49.650792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.650696    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc91222-888f-47a5-aa12-af6164fe2738-plugins-config-volume\") pod \"ccc91222-888f-47a5-aa12-af6164fe2738\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") "
Sep 17 23:39:49.650792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.650746    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtk4n\" (UniqueName: \"kubernetes.io/projected/ccc91222-888f-47a5-aa12-af6164fe2738-kube-api-access-jtk4n\") pod \"ccc91222-888f-47a5-aa12-af6164fe2738\" (UID: \"ccc91222-888f-47a5-aa12-af6164fe2738\") "
Sep 17 23:39:49.650792 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.650783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7d3d6c-7876-4c62-91d4-d81c83d40969-plugins-config-volume\") pod \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\" (UID: \"5c7d3d6c-7876-4c62-91d4-d81c83d40969\") "
Sep 17 23:39:49.651126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.651105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc91222-888f-47a5-aa12-af6164fe2738-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ccc91222-888f-47a5-aa12-af6164fe2738" (UID: "ccc91222-888f-47a5-aa12-af6164fe2738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:49.651188 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.651126    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7d3d6c-7876-4c62-91d4-d81c83d40969-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c7d3d6c-7876-4c62-91d4-d81c83d40969" (UID: "5c7d3d6c-7876-4c62-91d4-d81c83d40969"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:49.652941 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.652912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc91222-888f-47a5-aa12-af6164fe2738-kube-api-access-jtk4n" (OuterVolumeSpecName: "kube-api-access-jtk4n") pod "ccc91222-888f-47a5-aa12-af6164fe2738" (UID: "ccc91222-888f-47a5-aa12-af6164fe2738"). InnerVolumeSpecName "kube-api-access-jtk4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:49.652941 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.652927    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7d3d6c-7876-4c62-91d4-d81c83d40969-kube-api-access-b72sr" (OuterVolumeSpecName: "kube-api-access-b72sr") pod "5c7d3d6c-7876-4c62-91d4-d81c83d40969" (UID: "5c7d3d6c-7876-4c62-91d4-d81c83d40969"). InnerVolumeSpecName "kube-api-access-b72sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:49.751862 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.751841    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc91222-888f-47a5-aa12-af6164fe2738-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:49.751862 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.751862    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtk4n\" (UniqueName: \"kubernetes.io/projected/ccc91222-888f-47a5-aa12-af6164fe2738-kube-api-access-jtk4n\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:49.752034 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.751871    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7d3d6c-7876-4c62-91d4-d81c83d40969-plugins-config-volume\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:49.752034 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.751879    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b72sr\" (UniqueName: \"kubernetes.io/projected/5c7d3d6c-7876-4c62-91d4-d81c83d40969-kube-api-access-b72sr\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:39:49.782912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.782844    2559 generic.go:356] "Generic (PLEG): container finished" podID="ccc91222-888f-47a5-aa12-af6164fe2738" containerID="63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5" exitCode=0
Sep 17 23:39:49.782912 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.782889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"
Sep 17 23:39:49.783067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.782927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg" event={"ID":"ccc91222-888f-47a5-aa12-af6164fe2738","Type":"ContainerDied","Data":"63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5"}
Sep 17 23:39:49.783067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.782984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg" event={"ID":"ccc91222-888f-47a5-aa12-af6164fe2738","Type":"ContainerDied","Data":"4d31f2f0e98aa621729a6af7424d064a05c9c7522a41c456dd36052a0e940531"}
Sep 17 23:39:49.783067 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.783016    2559 scope.go:117] "RemoveContainer" containerID="63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5"
Sep 17 23:39:49.784197 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.784171    2559 generic.go:356] "Generic (PLEG): container finished" podID="5c7d3d6c-7876-4c62-91d4-d81c83d40969" containerID="8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729" exitCode=0
Sep 17 23:39:49.784296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.784216    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"
Sep 17 23:39:49.784296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.784261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s" event={"ID":"5c7d3d6c-7876-4c62-91d4-d81c83d40969","Type":"ContainerDied","Data":"8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729"}
Sep 17 23:39:49.784382 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.784313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s" event={"ID":"5c7d3d6c-7876-4c62-91d4-d81c83d40969","Type":"ContainerDied","Data":"30f92044b1148d801e5aa0a207f37869eba4499cca501ff613a5d09d4d7e03cf"}
Sep 17 23:39:49.792540 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.792522    2559 scope.go:117] "RemoveContainer" containerID="63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5"
Sep 17 23:39:49.792934 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:39:49.792788    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5\": container with ID starting with 63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5 not found: ID does not exist" containerID="63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5"
Sep 17 23:39:49.792934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.792815    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5"} err="failed to get container status \"63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5\": rpc error: code = NotFound desc = could not find container \"63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5\": container with ID starting with 63e5a6c22071dc3b3c8cfcb547bcde994d784c68e4cbe61d85724ddf8cac63a5 not found: ID does not exist"
Sep 17 23:39:49.792934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.792840    2559 scope.go:117] "RemoveContainer" containerID="8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729"
Sep 17 23:39:49.801120 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.801105    2559 scope.go:117] "RemoveContainer" containerID="8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729"
Sep 17 23:39:49.801352 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:39:49.801336    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729\": container with ID starting with 8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729 not found: ID does not exist" containerID="8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729"
Sep 17 23:39:49.801398 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.801357    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729"} err="failed to get container status \"8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729\": rpc error: code = NotFound desc = could not find container \"8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729\": container with ID starting with 8c53615c2151a68056c9379ed4f04897402bacd39cbf55e6e1c36c72d16b3729 not found: ID does not exist"
Sep 17 23:39:49.809870 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.809846    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"]
Sep 17 23:39:49.811446 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.811427    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d92920-6c69ffd77c-lh72s"]
Sep 17 23:39:49.823142 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.823120    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"]
Sep 17 23:39:49.834744 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:49.831131    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d92920-58b57c8598-fg8pg"]
Sep 17 23:39:51.639315 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:51.639279    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7d3d6c-7876-4c62-91d4-d81c83d40969" path="/var/lib/kubelet/pods/5c7d3d6c-7876-4c62-91d4-d81c83d40969/volumes"
Sep 17 23:39:51.639685 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:51.639613    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccc91222-888f-47a5-aa12-af6164fe2738" path="/var/lib/kubelet/pods/ccc91222-888f-47a5-aa12-af6164fe2738/volumes"
Sep 17 23:39:51.729060 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:51.729037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:39:59.997210 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:39:59.997173    2559 scope.go:117] "RemoveContainer" containerID="bc777696e778dee342f64609b98b62ad4f2e44120e35132623b07c615c1d8514"
Sep 17 23:40:00.620735 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:40:00.620693    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:40:59.743225 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:40:59.743202    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:40:59.748674 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:40:59.748651    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:40:59.751040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:40:59.751020    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:40:59.756074 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:40:59.756056    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:41:22.320568 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:41:22.320533    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:42:01.524702 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:42:01.524658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:42:09.226614 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:42:09.226582    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:42:16.329355 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:42:16.329322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:42:23.928247 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:42:23.928213    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:42:42.329313 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:42:42.329262    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:42:49.318856 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:42:49.318819    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:44:20.720732 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:20.720686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:44:26.424395 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:26.424357    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:44:31.718754 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.718707    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"]
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719061    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7d3d6c-7876-4c62-91d4-d81c83d40969" containerName="payload-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719073    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7d3d6c-7876-4c62-91d4-d81c83d40969" containerName="payload-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719086    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa04723-12bd-4525-b26f-f6e1d1841ccb" containerName="payload-pre-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719092    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa04723-12bd-4525-b26f-f6e1d1841ccb" containerName="payload-pre-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719101    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098d747f-2aa1-4ad9-8c51-50a40ae09a1e" containerName="payload-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719107    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="098d747f-2aa1-4ad9-8c51-50a40ae09a1e" containerName="payload-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719117    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" containerName="payload-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719122    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" containerName="payload-processing"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719127    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc78fef-4a8e-4deb-880a-526ee5b8c575" containerName="main"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719132    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc78fef-4a8e-4deb-880a-526ee5b8c575" containerName="main"
Sep 17 23:44:31.719137 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719140    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87e9df5-6d4e-40c4-8bc6-f513f29c001a" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719145    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87e9df5-6d4e-40c4-8bc6-f513f29c001a" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719153    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc91222-888f-47a5-aa12-af6164fe2738" containerName="payload-pre-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc91222-888f-47a5-aa12-af6164fe2738" containerName="payload-pre-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719165    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5292eec-5261-40a5-bce9-2afaf5c4660c" containerName="payload-pre-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719170    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5292eec-5261-40a5-bce9-2afaf5c4660c" containerName="payload-pre-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719175    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c24a3af-8573-4047-a8f3-b32b03d0d875" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719180    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c24a3af-8573-4047-a8f3-b32b03d0d875" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719231    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4e3a37-f35f-4054-a43c-3e0cc04dcb5c" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719238    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="098d747f-2aa1-4ad9-8c51-50a40ae09a1e" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719243    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc91222-888f-47a5-aa12-af6164fe2738" containerName="payload-pre-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719249    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7d3d6c-7876-4c62-91d4-d81c83d40969" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719254    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87e9df5-6d4e-40c4-8bc6-f513f29c001a" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719258    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c24a3af-8573-4047-a8f3-b32b03d0d875" containerName="payload-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719265    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc78fef-4a8e-4deb-880a-526ee5b8c575" containerName="main"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719271    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5292eec-5261-40a5-bce9-2afaf5c4660c" containerName="payload-pre-processing"
Sep 17 23:44:31.719514 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.719276    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa04723-12bd-4525-b26f-f6e1d1841ccb" containerName="payload-pre-processing"
Sep 17 23:44:31.722481 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.722461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.724830 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.724800    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d40f4868-kserve-self-signed-certs\""
Sep 17 23:44:31.732272 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.732252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"]
Sep 17 23:44:31.792177 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.792153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txd6\" (UniqueName: \"kubernetes.io/projected/1a40607f-df0c-4838-b108-b1c0a072a884-kube-api-access-9txd6\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.792268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.792187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-model-cache\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.792268 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.792206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-home\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.792340 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.792262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a40607f-df0c-4838-b108-b1c0a072a884-tls-certs\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.792340 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.792325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-dshm\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.792408 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.792365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-tmp-dir\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.893640 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.893607    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-tmp-dir\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.893757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.893670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txd6\" (UniqueName: \"kubernetes.io/projected/1a40607f-df0c-4838-b108-b1c0a072a884-kube-api-access-9txd6\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.893757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.893704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-model-cache\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.893757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.893752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-home\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.893876 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.893787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a40607f-df0c-4838-b108-b1c0a072a884-tls-certs\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.893876 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.893822    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-dshm\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.894049 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.894033    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-tmp-dir\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.894111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.894092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-model-cache\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.894111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.894102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-home\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.895814 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.895798    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-dshm\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.896178 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.896160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a40607f-df0c-4838-b108-b1c0a072a884-tls-certs\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:31.900256 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:31.900240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txd6\" (UniqueName: \"kubernetes.io/projected/1a40607f-df0c-4838-b108-b1c0a072a884-kube-api-access-9txd6\") pod \"e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") " pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:32.032311 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:32.032260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:32.300882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:32.300859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"]
Sep 17 23:44:32.305127 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:44:32.305067    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a40607f_df0c_4838_b108_b1c0a072a884.slice/crio-7d0d2b294da71ca3fc677fbee38e3e7c93596a449df457868a70dc1a9ee88899 WatchSource:0}: Error finding container 7d0d2b294da71ca3fc677fbee38e3e7c93596a449df457868a70dc1a9ee88899: Status 404 returned error can't find the container with id 7d0d2b294da71ca3fc677fbee38e3e7c93596a449df457868a70dc1a9ee88899
Sep 17 23:44:33.014156 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:33.014124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m" event={"ID":"1a40607f-df0c-4838-b108-b1c0a072a884","Type":"ContainerStarted","Data":"6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55"}
Sep 17 23:44:33.014492 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:33.014163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m" event={"ID":"1a40607f-df0c-4838-b108-b1c0a072a884","Type":"ContainerStarted","Data":"7d0d2b294da71ca3fc677fbee38e3e7c93596a449df457868a70dc1a9ee88899"}
Sep 17 23:44:33.030562 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:33.030519    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m" podStartSLOduration=2.030505188 podStartE2EDuration="2.030505188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:33.028956829 +0000 UTC m=+2315.502899826" watchObservedRunningTime="2026-09-17 23:44:33.030505188 +0000 UTC m=+2315.504448170"
Sep 17 23:44:34.734696 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:34.734664    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:44:42.032781 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:42.032695    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:42.032781 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:42.032755    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:42.044557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:42.044530    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:42.062937 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:42.062914    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:44:48.029680 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.029642    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"]
Sep 17 23:44:48.033422 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.033399    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.035584 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.035561    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d40f4868-kserve-self-signed-certs\""
Sep 17 23:44:48.042611 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.042587    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"]
Sep 17 23:44:48.115641 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.115618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-dshm\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.115770 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.115651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476d5d5-e49b-4b8f-b11f-9307676c763d-tls-certs\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.115770 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.115734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-model-cache\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.115865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.115789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-home\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.115865 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.115815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-tmp-dir\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.115978 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.115865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjncp\" (UniqueName: \"kubernetes.io/projected/b476d5d5-e49b-4b8f-b11f-9307676c763d-kube-api-access-fjncp\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.216896 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.216876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-home\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.216987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.216907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-tmp-dir\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.216987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.216932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjncp\" (UniqueName: \"kubernetes.io/projected/b476d5d5-e49b-4b8f-b11f-9307676c763d-kube-api-access-fjncp\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.216987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.216971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-dshm\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.216987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.216985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476d5d5-e49b-4b8f-b11f-9307676c763d-tls-certs\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.217173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.217011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-model-cache\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.217287 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.217266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-tmp-dir\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.217375 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.217357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-home\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.217428 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.217394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-model-cache\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.219173 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.219151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-dshm\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.219335 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.219319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476d5d5-e49b-4b8f-b11f-9307676c763d-tls-certs\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.224977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.224956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjncp\" (UniqueName: \"kubernetes.io/projected/b476d5d5-e49b-4b8f-b11f-9307676c763d-kube-api-access-fjncp\") pod \"e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") " pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.343626 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.343573    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:48.610891 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:48.610831    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"]
Sep 17 23:44:48.613053 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:44:48.613029    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb476d5d5_e49b_4b8f_b11f_9307676c763d.slice/crio-cbe1f6a0fa39647cb70d184dd587fcc43c457c3138afd9b38424612db54487da WatchSource:0}: Error finding container cbe1f6a0fa39647cb70d184dd587fcc43c457c3138afd9b38424612db54487da: Status 404 returned error can't find the container with id cbe1f6a0fa39647cb70d184dd587fcc43c457c3138afd9b38424612db54487da
Sep 17 23:44:49.083897 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:49.083869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb" event={"ID":"b476d5d5-e49b-4b8f-b11f-9307676c763d","Type":"ContainerStarted","Data":"dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c"}
Sep 17 23:44:49.084257 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:49.083905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb" event={"ID":"b476d5d5-e49b-4b8f-b11f-9307676c763d","Type":"ContainerStarted","Data":"cbe1f6a0fa39647cb70d184dd587fcc43c457c3138afd9b38424612db54487da"}
Sep 17 23:44:49.101027 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:49.100985    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb" podStartSLOduration=1.100970412 podStartE2EDuration="1.100970412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:49.099062798 +0000 UTC m=+2331.573005780" watchObservedRunningTime="2026-09-17 23:44:49.100970412 +0000 UTC m=+2331.574913426"
Sep 17 23:44:50.838105 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:50.838074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:44:58.343970 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:58.343938    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:58.344332 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:58.343983    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:58.355418 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:58.355394    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:44:59.138485 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:44:59.138461    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:45:19.852760 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:19.852707    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"]
Sep 17 23:45:21.417275 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:21.417245    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"]
Sep 17 23:45:22.332026 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:22.331989    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:45:30.926386 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:30.926348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:45:34.871815 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:34.871759    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m" podUID="1a40607f-df0c-4838-b108-b1c0a072a884" containerName="main" containerID="cri-o://6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55" gracePeriod=45
Sep 17 23:45:35.109525 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.109504    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:45:35.196741 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196678    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-tmp-dir\") pod \"1a40607f-df0c-4838-b108-b1c0a072a884\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") "
Sep 17 23:45:35.196741 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196737    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-home\") pod \"1a40607f-df0c-4838-b108-b1c0a072a884\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") "
Sep 17 23:45:35.196882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-model-cache\") pod \"1a40607f-df0c-4838-b108-b1c0a072a884\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") "
Sep 17 23:45:35.196882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a40607f-df0c-4838-b108-b1c0a072a884-tls-certs\") pod \"1a40607f-df0c-4838-b108-b1c0a072a884\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") "
Sep 17 23:45:35.196882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196797    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-dshm\") pod \"1a40607f-df0c-4838-b108-b1c0a072a884\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") "
Sep 17 23:45:35.196882 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196838    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9txd6\" (UniqueName: \"kubernetes.io/projected/1a40607f-df0c-4838-b108-b1c0a072a884-kube-api-access-9txd6\") pod \"1a40607f-df0c-4838-b108-b1c0a072a884\" (UID: \"1a40607f-df0c-4838-b108-b1c0a072a884\") "
Sep 17 23:45:35.197088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196929    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a40607f-df0c-4838-b108-b1c0a072a884" (UID: "1a40607f-df0c-4838-b108-b1c0a072a884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:35.197088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.196990    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a40607f-df0c-4838-b108-b1c0a072a884" (UID: "1a40607f-df0c-4838-b108-b1c0a072a884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:35.197088 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.197012    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-home" (OuterVolumeSpecName: "home") pod "1a40607f-df0c-4838-b108-b1c0a072a884" (UID: "1a40607f-df0c-4838-b108-b1c0a072a884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:35.197239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.197140    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:35.197239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.197158    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:35.197239 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.197166    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:35.198901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.198878    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-dshm" (OuterVolumeSpecName: "dshm") pod "1a40607f-df0c-4838-b108-b1c0a072a884" (UID: "1a40607f-df0c-4838-b108-b1c0a072a884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:35.198996 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.198962    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a40607f-df0c-4838-b108-b1c0a072a884-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a40607f-df0c-4838-b108-b1c0a072a884" (UID: "1a40607f-df0c-4838-b108-b1c0a072a884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:45:35.199056 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.198991    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a40607f-df0c-4838-b108-b1c0a072a884-kube-api-access-9txd6" (OuterVolumeSpecName: "kube-api-access-9txd6") pod "1a40607f-df0c-4838-b108-b1c0a072a884" (UID: "1a40607f-df0c-4838-b108-b1c0a072a884"). InnerVolumeSpecName "kube-api-access-9txd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:45:35.295551 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.295524    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a40607f-df0c-4838-b108-b1c0a072a884" containerID="6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55" exitCode=0
Sep 17 23:45:35.295645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.295577    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"
Sep 17 23:45:35.295645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.295581    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m" event={"ID":"1a40607f-df0c-4838-b108-b1c0a072a884","Type":"ContainerDied","Data":"6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55"}
Sep 17 23:45:35.295645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.295619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m" event={"ID":"1a40607f-df0c-4838-b108-b1c0a072a884","Type":"ContainerDied","Data":"7d0d2b294da71ca3fc677fbee38e3e7c93596a449df457868a70dc1a9ee88899"}
Sep 17 23:45:35.295645 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.295639    2559 scope.go:117] "RemoveContainer" containerID="6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55"
Sep 17 23:45:35.297535 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.297518    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a40607f-df0c-4838-b108-b1c0a072a884-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:35.297535 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.297534    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a40607f-df0c-4838-b108-b1c0a072a884-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:35.297682 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.297542    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txd6\" (UniqueName: \"kubernetes.io/projected/1a40607f-df0c-4838-b108-b1c0a072a884-kube-api-access-9txd6\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:35.304852 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.304833    2559 scope.go:117] "RemoveContainer" containerID="6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55"
Sep 17 23:45:35.305087 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:45:35.305070    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55\": container with ID starting with 6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55 not found: ID does not exist" containerID="6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55"
Sep 17 23:45:35.305126 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.305094    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55"} err="failed to get container status \"6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55\": rpc error: code = NotFound desc = could not find container \"6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55\": container with ID starting with 6a0d45ae51a36f9c7336ded0172765f31026140d206fff83883c0537c9690d55 not found: ID does not exist"
Sep 17 23:45:35.316837 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.316818    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"]
Sep 17 23:45:35.319901 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.319881    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d40f4868-kserve-76486775c5-xgf9m"]
Sep 17 23:45:35.639311 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:35.639287    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a40607f-df0c-4838-b108-b1c0a072a884" path="/var/lib/kubelet/pods/1a40607f-df0c-4838-b108-b1c0a072a884/volumes"
Sep 17 23:45:36.435557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.435525    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb" podUID="b476d5d5-e49b-4b8f-b11f-9307676c763d" containerName="main" containerID="cri-o://dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c" gracePeriod=45
Sep 17 23:45:36.674685 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.674663    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:45:36.807854 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.807814    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjncp\" (UniqueName: \"kubernetes.io/projected/b476d5d5-e49b-4b8f-b11f-9307676c763d-kube-api-access-fjncp\") pod \"b476d5d5-e49b-4b8f-b11f-9307676c763d\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") "
Sep 17 23:45:36.807987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.807872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476d5d5-e49b-4b8f-b11f-9307676c763d-tls-certs\") pod \"b476d5d5-e49b-4b8f-b11f-9307676c763d\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") "
Sep 17 23:45:36.807987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.807900    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-dshm\") pod \"b476d5d5-e49b-4b8f-b11f-9307676c763d\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") "
Sep 17 23:45:36.807987 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.807935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-home\") pod \"b476d5d5-e49b-4b8f-b11f-9307676c763d\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") "
Sep 17 23:45:36.808154 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.808004    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-model-cache\") pod \"b476d5d5-e49b-4b8f-b11f-9307676c763d\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") "
Sep 17 23:45:36.808154 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.808038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-tmp-dir\") pod \"b476d5d5-e49b-4b8f-b11f-9307676c763d\" (UID: \"b476d5d5-e49b-4b8f-b11f-9307676c763d\") "
Sep 17 23:45:36.808296 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.808265    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b476d5d5-e49b-4b8f-b11f-9307676c763d" (UID: "b476d5d5-e49b-4b8f-b11f-9307676c763d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:36.808350 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.808276    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-home" (OuterVolumeSpecName: "home") pod "b476d5d5-e49b-4b8f-b11f-9307676c763d" (UID: "b476d5d5-e49b-4b8f-b11f-9307676c763d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:36.808461 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.808422    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b476d5d5-e49b-4b8f-b11f-9307676c763d" (UID: "b476d5d5-e49b-4b8f-b11f-9307676c763d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:36.809968 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.809946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-dshm" (OuterVolumeSpecName: "dshm") pod "b476d5d5-e49b-4b8f-b11f-9307676c763d" (UID: "b476d5d5-e49b-4b8f-b11f-9307676c763d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:36.810060 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.810043    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b476d5d5-e49b-4b8f-b11f-9307676c763d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b476d5d5-e49b-4b8f-b11f-9307676c763d" (UID: "b476d5d5-e49b-4b8f-b11f-9307676c763d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:45:36.810151 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.810135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b476d5d5-e49b-4b8f-b11f-9307676c763d-kube-api-access-fjncp" (OuterVolumeSpecName: "kube-api-access-fjncp") pod "b476d5d5-e49b-4b8f-b11f-9307676c763d" (UID: "b476d5d5-e49b-4b8f-b11f-9307676c763d"). InnerVolumeSpecName "kube-api-access-fjncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:45:36.908960 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.908932    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjncp\" (UniqueName: \"kubernetes.io/projected/b476d5d5-e49b-4b8f-b11f-9307676c763d-kube-api-access-fjncp\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:36.908960 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.908955    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476d5d5-e49b-4b8f-b11f-9307676c763d-tls-certs\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:36.909103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.908968    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-dshm\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:36.909103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.908980    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-home\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:36.909103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.908991    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-model-cache\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:36.909103 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:36.909002    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476d5d5-e49b-4b8f-b11f-9307676c763d-tmp-dir\") on node \"ip-10-0-143-250.ec2.internal\" DevicePath \"\""
Sep 17 23:45:37.306619 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.306595    2559 generic.go:356] "Generic (PLEG): container finished" podID="b476d5d5-e49b-4b8f-b11f-9307676c763d" containerID="dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c" exitCode=0
Sep 17 23:45:37.306758 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.306629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb" event={"ID":"b476d5d5-e49b-4b8f-b11f-9307676c763d","Type":"ContainerDied","Data":"dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c"}
Sep 17 23:45:37.306758 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.306637    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"
Sep 17 23:45:37.306758 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.306656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb" event={"ID":"b476d5d5-e49b-4b8f-b11f-9307676c763d","Type":"ContainerDied","Data":"cbe1f6a0fa39647cb70d184dd587fcc43c457c3138afd9b38424612db54487da"}
Sep 17 23:45:37.306758 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.306672    2559 scope.go:117] "RemoveContainer" containerID="dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c"
Sep 17 23:45:37.315557 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.315542    2559 scope.go:117] "RemoveContainer" containerID="dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c"
Sep 17 23:45:37.315812 ip-10-0-143-250 kubenswrapper[2559]: E0917 23:45:37.315794    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c\": container with ID starting with dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c not found: ID does not exist" containerID="dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c"
Sep 17 23:45:37.315887 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.315825    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c"} err="failed to get container status \"dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c\": rpc error: code = NotFound desc = could not find container \"dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c\": container with ID starting with dce63b40b95c945a26a7448f6b7db503e7abf25583e038e5399bc850693a399c not found: ID does not exist"
Sep 17 23:45:37.326526 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.326503    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"]
Sep 17 23:45:37.329773 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.329754    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d40f4868-kserve-564887d75-wpmcb"]
Sep 17 23:45:37.639888 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:37.639827    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b476d5d5-e49b-4b8f-b11f-9307676c763d" path="/var/lib/kubelet/pods/b476d5d5-e49b-4b8f-b11f-9307676c763d/volumes"
Sep 17 23:45:40.225754 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:40.225706    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:45:48.027241 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:48.027204    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:45:54.933402 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:54.933366    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:45:59.772561 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:59.772531    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:45:59.780380 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:59.780355    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:45:59.780757 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:59.780735    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:45:59.788009 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:45:59.787993    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nq45x_7e63ef48-0367-4e32-b7af-f2f23c7dd048/ovn-acl-logging/0.log"
Sep 17 23:46:00.519451 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:00.519426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:46:07.227867 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:07.227837    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qq8l2"]
Sep 17 23:46:16.853291 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:16.853184    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-c8f2c_2ddca1d2-1117-442d-a537-26515c0cacab/copy-manifests/0.log"
Sep 17 23:46:16.859395 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:16.859377    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-c8f2c_2ddca1d2-1117-442d-a537-26515c0cacab/manager/0.log"
Sep 17 23:46:19.061396 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:19.061368    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pjlms_4fc0bfad-448c-4b7b-adf9-91fabb127593/kuadrant-console-plugin/0.log"
Sep 17 23:46:19.318372 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:19.318313    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qq8l2_3b4115b4-71b1-46d2-be35-c16dfdb14a60/limitador/0.log"
Sep 17 23:46:20.038753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:20.038713    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-b7wws_4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6/kube-auth-proxy/0.log"
Sep 17 23:46:20.390825 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:20.390769    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-65bws_373051c0-27e6-4947-ab58-e13b1a9182e4/payload-pre-processing/0.log"
Sep 17 23:46:20.641655 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:20.641613    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f8ffff48-x67qv_ceed8fa5-da35-4ace-a55b-428695ec6af8/router/0.log"
Sep 17 23:46:21.223767 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:21.223749    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-7hw79_e0b26bfa-6959-4baf-ad76-89ecd0431c13/main/0.log"
Sep 17 23:46:21.580347 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:21.580293    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-kww9r_22feb1fb-5506-49fb-8878-3a6104a0a0e2/main/0.log"
Sep 17 23:46:21.693166 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:21.693145    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-svp2b_ad583cf3-dc68-4db9-930e-8fe032107dd8/main/0.log"
Sep 17 23:46:30.718610 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:30.718580    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4qpfx_54dd74cf-a6ef-4276-addf-456e170bc225/global-pull-secret-syncer/0.log"
Sep 17 23:46:30.864359 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:30.864337    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sb77f_621e2e6d-aeaa-4650-b04f-8dbba3a8e30b/konnectivity-agent/0.log"
Sep 17 23:46:30.941646 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:30.941614    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-250.ec2.internal_55c41d7938cdbb09331b64496abe4d6c/haproxy/0.log"
Sep 17 23:46:35.478892 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:35.478871    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pjlms_4fc0bfad-448c-4b7b-adf9-91fabb127593/kuadrant-console-plugin/0.log"
Sep 17 23:46:35.539763 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:35.539735    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qq8l2_3b4115b4-71b1-46d2-be35-c16dfdb14a60/limitador/0.log"
Sep 17 23:46:37.225979 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.225951    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-xxsct_7ea26fcd-6335-471b-8e55-07d32f9b6ff0/cluster-monitoring-operator/0.log"
Sep 17 23:46:37.448923 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.448898    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqgpc_d353d6cb-d949-4cdf-b425-0b1df26f10ea/node-exporter/0.log"
Sep 17 23:46:37.470052 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.470029    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqgpc_d353d6cb-d949-4cdf-b425-0b1df26f10ea/kube-rbac-proxy/0.log"
Sep 17 23:46:37.494404 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.494386    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqgpc_d353d6cb-d949-4cdf-b425-0b1df26f10ea/init-textfile/0.log"
Sep 17 23:46:37.715365 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.715336    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-s4jt4_f47eb749-c30c-4f90-aa6c-1bd247fca6b3/kube-rbac-proxy-main/0.log"
Sep 17 23:46:37.739412 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.739389    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-s4jt4_f47eb749-c30c-4f90-aa6c-1bd247fca6b3/kube-rbac-proxy-self/0.log"
Sep 17 23:46:37.760677 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.760652    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-s4jt4_f47eb749-c30c-4f90-aa6c-1bd247fca6b3/openshift-state-metrics/0.log"
Sep 17 23:46:37.802224 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.802199    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/prometheus/0.log"
Sep 17 23:46:37.826456 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.826434    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/config-reloader/0.log"
Sep 17 23:46:37.849977 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.849950    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/thanos-sidecar/0.log"
Sep 17 23:46:37.870995 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.870969    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/kube-rbac-proxy-web/0.log"
Sep 17 23:46:37.892235 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.892218    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/kube-rbac-proxy/0.log"
Sep 17 23:46:37.912193 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.912177    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/kube-rbac-proxy-thanos/0.log"
Sep 17 23:46:37.943689 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.943657    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db953bb-c30d-42f2-a534-d048ee960b12/init-config-reloader/0.log"
Sep 17 23:46:37.973593 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.973535    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-rlbkf_42a1c7f2-ca1b-4c18-ae98-cf9e40209076/prometheus-operator/0.log"
Sep 17 23:46:37.992600 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:37.992578    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-rlbkf_42a1c7f2-ca1b-4c18-ae98-cf9e40209076/kube-rbac-proxy/0.log"
Sep 17 23:46:38.016455 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:38.016436    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-c5c66_a948559c-749d-4abd-8979-b49885584561/prometheus-operator-admission-webhook/0.log"
Sep 17 23:46:38.045404 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:38.045382    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cbb4fb674-vxlkj_c49c25c8-b81c-4e02-9195-7a3b7a93e689/telemeter-client/0.log"
Sep 17 23:46:38.065504 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:38.065481    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cbb4fb674-vxlkj_c49c25c8-b81c-4e02-9195-7a3b7a93e689/reload/0.log"
Sep 17 23:46:38.084168 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:38.084147    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cbb4fb674-vxlkj_c49c25c8-b81c-4e02-9195-7a3b7a93e689/kube-rbac-proxy/0.log"
Sep 17 23:46:39.588622 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:39.588594    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-qhhvx_ef4cf7f0-9ac6-4e3a-85b7-d5892828e9d1/networking-console-plugin/0.log"
Sep 17 23:46:40.174855 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.174817    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/1.log"
Sep 17 23:46:40.179197 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.179177    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k8f5r_06ef5388-8c10-4002-a3ae-1ddb404a54e7/console-operator/2.log"
Sep 17 23:46:40.620516 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620479    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"]
Sep 17 23:46:40.620975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620879    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a40607f-df0c-4838-b108-b1c0a072a884" containerName="main"
Sep 17 23:46:40.620975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620893    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a40607f-df0c-4838-b108-b1c0a072a884" containerName="main"
Sep 17 23:46:40.620975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620902    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b476d5d5-e49b-4b8f-b11f-9307676c763d" containerName="main"
Sep 17 23:46:40.620975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620907    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b476d5d5-e49b-4b8f-b11f-9307676c763d" containerName="main"
Sep 17 23:46:40.620975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620969    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a40607f-df0c-4838-b108-b1c0a072a884" containerName="main"
Sep 17 23:46:40.620975 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.620976    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b476d5d5-e49b-4b8f-b11f-9307676c763d" containerName="main"
Sep 17 23:46:40.624098 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.624083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.626609 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.626586    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kg5sm\"/\"openshift-service-ca.crt\""
Sep 17 23:46:40.627403 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.627379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kg5sm\"/\"default-dockercfg-hwp27\""
Sep 17 23:46:40.627403 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.627395    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kg5sm\"/\"kube-root-ca.crt\""
Sep 17 23:46:40.633753 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.633701    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"]
Sep 17 23:46:40.689953 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.689919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkl4\" (UniqueName: \"kubernetes.io/projected/3bf7733f-f362-4f01-92e7-0490c403a7a2-kube-api-access-tdkl4\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.690117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.689981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-podres\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.690117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.690019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-lib-modules\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.690117 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.690068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-sys\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.690273 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.690130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-proc\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791081 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkl4\" (UniqueName: \"kubernetes.io/projected/3bf7733f-f362-4f01-92e7-0490c403a7a2-kube-api-access-tdkl4\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-podres\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-lib-modules\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-sys\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-proc\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791280 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-sys\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791483 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-podres\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791483 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-proc\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.791483 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.791305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bf7733f-f362-4f01-92e7-0490c403a7a2-lib-modules\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.798452 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.798434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkl4\" (UniqueName: \"kubernetes.io/projected/3bf7733f-f362-4f01-92e7-0490c403a7a2-kube-api-access-tdkl4\") pod \"perf-node-gather-daemonset-pfdk7\" (UID: \"3bf7733f-f362-4f01-92e7-0490c403a7a2\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:40.935258 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:40.935179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:41.204150 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:41.204124    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"]
Sep 17 23:46:41.206294 ip-10-0-143-250 kubenswrapper[2559]: W0917 23:46:41.206265    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3bf7733f_f362_4f01_92e7_0490c403a7a2.slice/crio-c9bef88ca8a5974b44f6bc69d66a4a880291a8d740e6ddfcd0fe2703e83b11a7 WatchSource:0}: Error finding container c9bef88ca8a5974b44f6bc69d66a4a880291a8d740e6ddfcd0fe2703e83b11a7: Status 404 returned error can't find the container with id c9bef88ca8a5974b44f6bc69d66a4a880291a8d740e6ddfcd0fe2703e83b11a7
Sep 17 23:46:41.586966 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:41.586875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7" event={"ID":"3bf7733f-f362-4f01-92e7-0490c403a7a2","Type":"ContainerStarted","Data":"53cc6158879fbafb71bd47bdbb65542435742dab023f66b3a00d0742a3509c01"}
Sep 17 23:46:41.586966 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:41.586937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7" event={"ID":"3bf7733f-f362-4f01-92e7-0490c403a7a2","Type":"ContainerStarted","Data":"c9bef88ca8a5974b44f6bc69d66a4a880291a8d740e6ddfcd0fe2703e83b11a7"}
Sep 17 23:46:41.587145 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:41.587038    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:41.602948 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:41.602905    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7" podStartSLOduration=1.602890394 podStartE2EDuration="1.602890394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:41.600464752 +0000 UTC m=+2444.074407726" watchObservedRunningTime="2026-09-17 23:46:41.602890394 +0000 UTC m=+2444.076833376"
Sep 17 23:46:42.490454 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:42.490414    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4vqv_0f20abf2-d57f-4040-bb02-94cd0beb80fb/dns/0.log"
Sep 17 23:46:42.509964 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:42.509941    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4vqv_0f20abf2-d57f-4040-bb02-94cd0beb80fb/kube-rbac-proxy/0.log"
Sep 17 23:46:42.617012 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:42.616985    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c6w6w_16511250-0c7e-4b12-841b-afeba2af4bc8/dns-node-resolver/0.log"
Sep 17 23:46:43.236772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:43.236742    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d98wj_af2fe43e-8f00-4eeb-b27d-1ff01d211599/node-ca/0.log"
Sep 17 23:46:44.538772 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:44.538743    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-b7wws_4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6/kube-auth-proxy/0.log"
Sep 17 23:46:44.621496 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:44.621469    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-65bws_373051c0-27e6-4947-ab58-e13b1a9182e4/payload-pre-processing/0.log"
Sep 17 23:46:44.677040 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:44.677017    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f8ffff48-x67qv_ceed8fa5-da35-4ace-a55b-428695ec6af8/router/0.log"
Sep 17 23:46:45.529897 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:45.529865    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnpbs_e42d7a0c-f350-4657-b80b-f024df6affb0/serve-healthcheck-canary/0.log"
Sep 17 23:46:45.979316 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:45.979290    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-x5n54_fe98b390-a870-4ae3-9656-cb7d1dc56c29/insights-operator/0.log"
Sep 17 23:46:45.979711 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:45.979428    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-x5n54_fe98b390-a870-4ae3-9656-cb7d1dc56c29/insights-operator/1.log"
Sep 17 23:46:46.063301 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:46.063278    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-58qvv_88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c/kube-rbac-proxy/0.log"
Sep 17 23:46:46.082111 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:46.082090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-58qvv_88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c/exporter/0.log"
Sep 17 23:46:46.102179 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:46.102155    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-58qvv_88e83fe1-e0b8-4ee6-baa5-aa8e74f7797c/extractor/0.log"
Sep 17 23:46:47.601634 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:47.601605    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-pfdk7"
Sep 17 23:46:48.666934 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:48.666903    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-c8f2c_2ddca1d2-1117-442d-a537-26515c0cacab/manager/0.log"
Sep 17 23:46:48.690219 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:48.690195    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-c8f2c_2ddca1d2-1117-442d-a537-26515c0cacab/copy-manifests/0.log"
Sep 17 23:46:50.228589 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:50.228556    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-678fd6f85-dcq2d_546b24b5-7de7-4e83-8045-897fc2757973/manager/0.log"
Sep 17 23:46:50.282207 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:50.282139    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6hg4n_9edc9f7f-6817-45fd-8de5-ddff0ae1b59f/openshift-lws-operator/0.log"
Sep 17 23:46:51.171839 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:51.171809    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-b7wws_4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6/kube-auth-proxy/0.log"
Sep 17 23:46:51.227883 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:51.227854    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-65bws_373051c0-27e6-4947-ab58-e13b1a9182e4/payload-pre-processing/0.log"
Sep 17 23:46:51.270816 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:51.270792    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f8ffff48-x67qv_ceed8fa5-da35-4ace-a55b-428695ec6af8/router/0.log"
Sep 17 23:46:53.105385 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:53.105361    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-b7wws_4ab03d10-ee8b-45ac-ae6f-d2a2a7283ec6/kube-auth-proxy/0.log"
Sep 17 23:46:53.168393 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:53.168365    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-65bws_373051c0-27e6-4947-ab58-e13b1a9182e4/payload-pre-processing/0.log"
Sep 17 23:46:53.219615 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:53.219591    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77f8ffff48-x67qv_ceed8fa5-da35-4ace-a55b-428695ec6af8/router/0.log"
Sep 17 23:46:56.851284 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:56.851259    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:46:56.879544 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:56.879519    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/egress-router-binary-copy/0.log"
Sep 17 23:46:56.912952 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:56.912934    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/cni-plugins/0.log"
Sep 17 23:46:56.941254 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:56.941236    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/bond-cni-plugin/0.log"
Sep 17 23:46:56.967442 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:56.967427    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/routeoverride-cni/0.log"
Sep 17 23:46:56.993122 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:56.993103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/whereabouts-cni-bincopy/0.log"
Sep 17 23:46:57.017231 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:57.017212    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2bsm_647f02eb-3c4a-4b27-8d3a-9892a08754df/whereabouts-cni/0.log"
Sep 17 23:46:57.249709 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:57.249682    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sxdqn_86c50601-fa82-4f6a-a9f0-a645bf3000c6/kube-multus/0.log"
Sep 17 23:46:57.371620 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:57.371595    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kd746_4404befb-c49e-440b-9f60-9b04e47390d0/network-metrics-daemon/0.log"
Sep 17 23:46:57.388176 ip-10-0-143-250 kubenswrapper[2559]: I0917 23:46:57.388159    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kd746_4404befb-c49e-440b-9f60-9b04e47390d0/kube-rbac-proxy/0.log"