Sep 08 15:15:38.900186 ip-10-0-142-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 15:15:38.900199 ip-10-0-142-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 15:15:38.900206 ip-10-0-142-52 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 15:15:38.900427 ip-10-0-142-52 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 15:15:49.149369 ip-10-0-142-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 15:15:49.149386 ip-10-0-142-52 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 690368c8699e41d293c2262e5b4c476f --
Sep 08 15:17:30.812128 ip-10-0-142-52 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 15:17:31.209468 ip-10-0-142-52 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:17:31.209468 ip-10-0-142-52 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 15:17:31.209468 ip-10-0-142-52 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:17:31.209468 ip-10-0-142-52 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 15:17:31.209468 ip-10-0-142-52 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:17:31.210137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.209534    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 15:17:31.215335 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215319    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:17:31.215335 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215334    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215338    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215341    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215344    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215347    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215350    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215353    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215356    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215358    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215361    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215364    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215366    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215369    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215372    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215374    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215377    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215379    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215382    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215384    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215387    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:17:31.215402 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215389    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215391    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215394    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215398    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215402    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215405    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215408    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215411    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215414    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215417    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215420    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215423    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215426    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215429    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215432    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215434    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215437    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215441    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215443    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:17:31.215906 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215447    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215450    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215454    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215456    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215459    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215462    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215465    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215468    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215470    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215473    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215476    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215478    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215481    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215484    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215486    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215490    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215492    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215495    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215497    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215500    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:17:31.216397 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215503    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215505    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215508    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215511    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215513    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215515    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215518    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215520    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215523    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215525    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215528    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215530    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215533    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215535    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215538    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215540    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215543    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215546    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215548    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215550    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:17:31.217265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215553    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215555    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215558    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215560    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215563    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215565    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215568    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215571    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215574    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215576    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215579    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215581    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215585    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215589    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215592    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215594    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215597    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215600    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.215602    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:17:31.217948 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216270    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216279    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216282    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216286    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216288    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216291    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216293    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216296    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216298    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216301    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216303    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216306    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216308    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216311    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216313    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216316    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216318    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216320    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216323    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:17:31.218547 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216326    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216329    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216331    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216334    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216337    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216339    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216341    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216344    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216378    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216562    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216570    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216574    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216579    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216584    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216588    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216593    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216598    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216602    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216606    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:17:31.219084 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216611    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216616    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216620    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216629    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216635    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216639    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216643    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216647    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216653    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216658    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216662    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216666    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216670    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216674    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216683    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216688    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216692    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216696    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216700    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216705    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:17:31.219557 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216709    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216713    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216728    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216733    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216737    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216744    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216755    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216759    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216764    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216769    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216773    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216777    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216781    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216786    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216790    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216794    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216798    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216802    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216807    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216815    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:17:31.220074 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216819    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216823    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216828    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216835    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216839    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216861    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216867    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216873    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216879    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216883    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216889    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216898    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216902    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216906    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.216911    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.217324    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.217330    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.217334    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.217339    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:17:31.220548 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.217343    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.217348    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218321    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218332    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218336    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218339    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218342    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218344    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218347    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218350    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218353    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218356    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218359    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218362    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218364    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218367    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218369    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218372    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218375    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218377    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:17:31.221029 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218380    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218382    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218385    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218388    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218390    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218392    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218395    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218398    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218401    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218403    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218406    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218410    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218414    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218417    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218420    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218424    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218428    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218431    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:17:31.221510 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218434    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218437    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218439    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218442    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218444    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218447    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218449    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218452    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218454    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218457    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218460    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218464    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218468    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218470    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218473    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218475    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218478    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218480    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218483    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218485    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:17:31.221972 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218488    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218490    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218493    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218495    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218498    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218500    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218503    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218505    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218508    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218513    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218516    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218518    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218521    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218525    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218527    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218530    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218532    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218535    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218537    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218540    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:17:31.222489 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218542    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218544    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218547    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218550    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218553    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218555    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218558    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218560    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218563    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218566    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218568    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218571    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218573    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218576    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218578    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218581    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218583    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218585    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218588    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218590    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:17:31.223006 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218593    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218597    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.218600    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218670    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218704    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218710    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218714    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218718    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218721    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218726    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218730    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218733    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218735    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218740    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218743    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218746    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218749    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218752    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218754    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218757    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218760    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218764    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218767    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218770    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 15:17:31.223492 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218773    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218776    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218780    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218783    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218786    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218789    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218792    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218795    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218797    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218802    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218805    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218809    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218812    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218815    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218818    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218821    2575 flags.go:64] FLAG: --enable-server="true"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218824    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218828    2575 flags.go:64] FLAG: --event-burst="100"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218831    2575 flags.go:64] FLAG: --event-qps="50"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218834    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218837    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218840    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218864    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218867    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218870    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 15:17:31.224152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218873    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218876    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218878    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218881    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218884    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218887    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218889    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218892    2575 flags.go:64] FLAG: --feature-gates=""
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218895    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218898    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218901    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218904    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218906    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218909    2575 flags.go:64] FLAG: --help="false"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218912    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218914    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218919    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218921    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218924    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218928    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218930    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218933    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218935    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 15:17:31.224758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218938    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218941    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218944    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218946    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218949    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218951    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218954    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218957    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218960    2575 flags.go:64] FLAG: --lock-file=""
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218963    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218965    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218968    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218973    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218976    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218978    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218981    2575 flags.go:64] FLAG: --logging-format="text"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218984    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218987    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218989    2575 flags.go:64] FLAG: --manifest-url=""
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218992    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218996    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.218999    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219002    2575 flags.go:64] FLAG: --max-pods="110"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219005    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219008    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 15:17:31.225365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219012    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219015    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219017    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219020    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219023    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219029    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219032    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219035    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219038    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219040    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219046    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219059    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219062    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219065    2575 flags.go:64] FLAG: --port="10250"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219068    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219071    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027ca8b6a697722b7"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219074    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219077    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219080    2575 flags.go:64] FLAG: --register-node="true"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219082    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219091    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219094    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219096    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219099    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219102    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 15:17:31.226051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219105    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219108    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219111    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219113    2575 flags.go:64] FLAG: --runonce="false"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219116    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219119    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219121    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219126    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219130    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219133    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219136    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219139    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219142    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219145    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219147    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219150    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219153    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219156    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219158    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219164    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219167    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219169    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219173    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219176    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219182    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 15:17:31.226681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219185    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219188    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219190    2575 flags.go:64] FLAG: --v="2"
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219194    2575 flags.go:64] FLAG: --version="false"
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219198    2575 flags.go:64] FLAG: --vmodule=""
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219201    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.219204    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219310    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219314    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219317    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219320    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219322    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219325    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219328    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219336    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219340    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219343    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219346    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219349    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219351    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219354    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:17:31.227310 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219356    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219359    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219361    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219363    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219366    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219368    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219371    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219373    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219376    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219378    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219382    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219384    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219387    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219389    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219392    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219394    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219396    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219399    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219401    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219404    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:17:31.227821 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219406    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219408    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219411    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219413    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219416    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219420    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219424    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219426    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219430    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219434    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219437    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219439    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219442    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219444    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219447    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219449    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219451    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219454    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219457    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:17:31.228395 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219461    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219464    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219467    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219470    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219473    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219476    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219478    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219482    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219485    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219488    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219491    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219493    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219496    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219499    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219501    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219504    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219506    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219509    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219512    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:17:31.228871 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219517    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219519    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219522    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219524    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219527    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219530    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219532    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219534    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219537    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219539    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219542    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219544    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219547    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219549    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219552    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219554    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219556    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219559    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219562    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219565    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:17:31.229330 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219567    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219570    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219572    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219575    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219577    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219579    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.219582    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.220810    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.227868    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.227881    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.228632    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.228639    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228691    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228696    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228699    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:17:31.229879 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228703    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228705    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228708    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228711    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228714    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228716    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228719    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228721    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228724    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228726    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228729    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228731    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228734    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228737    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228739    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228742    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228744    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228746    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228749    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228752    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:17:31.230255 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228755    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228757    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228760    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228762    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228765    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228768    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228772    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228775    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228777    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228780    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228782    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228785    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228787    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228790    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228792    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228795    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228797    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228800    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228802    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:17:31.230736 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228805    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228808    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228810    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228812    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228815    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228819    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228822    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228824    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228827    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228830    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228833    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228835    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228838    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228855    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228862    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228867    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228870    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228874    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228877    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228880    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:17:31.231201 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228883    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228886    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228889    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228892    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228895    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228897    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228900    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228903    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228905    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228908    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228911    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228913    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228915    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228918    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228921    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228923    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228926    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228928    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228931    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228933    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:17:31.231677 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228936    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228938    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228941    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228943    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228946    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228948    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228951    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228953    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228956    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228958    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228961    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228965    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228969    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228972    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228975    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228977    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:17:31.232172 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.228980    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.228986    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229094    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229099    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229102    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229105    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229108    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229110    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229113    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229115    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229118    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229120    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229123    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229125    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:17:31.232554 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229127    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229130    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229132    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229135    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229137    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229139    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229142    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229144    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229147    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229149    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229152    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229154    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229157    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229160    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229163    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229165    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229172    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229174    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229177    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229179    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:17:31.232913 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229182    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229184    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229187    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229189    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229191    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229194    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229196    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229199    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229202    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229204    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229206    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229209    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229211    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229214    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229216    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229219    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229221    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229224    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229226    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229229    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:17:31.233394 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229231    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229233    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229236    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229238    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229241    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229243    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229246    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229248    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229251    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229254    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229257    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229259    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229262    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229264    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229267    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229269    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229271    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229274    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229276    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229279    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:17:31.233891 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229281    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229284    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229286    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229289    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229293    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229297    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229301    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229303    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229307    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229311    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229314    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229316    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229319    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229321    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229324    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229326    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229328    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229331    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:17:31.234362 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229334    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229336    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229339    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229342    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229345    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229348    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229350    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229353    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:31.229355    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.229360    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.229463    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.231603    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.232428    2575 server.go:1019] "Starting client certificate rotation"
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.232532    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:17:31.234799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.233182    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:17:31.235185 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.234970    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:17:31.238612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.238592    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:17:31.252083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.252067    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 08 15:17:31.255007 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.254990    2575 log.go:25] "Validated CRI v1 image API"
Sep 08 15:17:31.255831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.255814    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 15:17:31.259793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.259772    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7db3c5b8-1581-45cb-954a-7f7d3c639181:/dev/nvme0n1p3 7ea70d13-8f57-49f7-a9c9-49f5d5d2827b:/dev/nvme0n1p4]
Sep 08 15:17:31.259866 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.259794    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 15:17:31.260996 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.260981    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:17:31.265214 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.265109    2575 manager.go:216] Machine: {Timestamp:2026-09-08 15:17:31.263696997 +0000 UTC m=+0.338989301 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097506 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec234b477f02576d857d306719fb1e5a SystemUUID:ec234b47-7f02-576d-857d-306719fb1e5a BootID:690368c8-699e-41d2-93c2-262e5b4c476f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cb:61:5d:45:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cb:61:5d:45:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:90:26:be:5d:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 15:17:31.265214 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.265213    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 15:17:31.265340 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.265283    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 15:17:31.268132 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.268107    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 15:17:31.268257 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.268136    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 15:17:31.268874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.268864    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 15:17:31.268909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.268876    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 15:17:31.268909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.268884    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:17:31.269644 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.269635    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:17:31.270870 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.270860    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 15:17:31.274034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274023    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:17:31.274152 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274143    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 15:17:31.274181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274171    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 15:17:31.274181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274180    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 15:17:31.274236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274190    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 15:17:31.274236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274199    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 08 15:17:31.274236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274207    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 15:17:31.274491 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274479    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:17:31.274531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.274500    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:17:31.276793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.276775    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 15:17:31.279603 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279590    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 15:17:31.279678 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279620    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 15:17:31.279761 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279752    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 15:17:31.279794 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279768    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 15:17:31.279794 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279775    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 15:17:31.279794 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279781    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 15:17:31.279794 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279788    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 15:17:31.279914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.279801    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 15:17:31.280218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.280209    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 15:17:31.280248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.280218    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 15:17:31.280248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.280225    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 15:17:31.280248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.280231    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 15:17:31.280248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.280240    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 15:17:31.280248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.280247    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 15:17:31.281179 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.281164    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 15:17:31.281179 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.281178    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 15:17:31.282940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.282912    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f6c8"
Sep 08 15:17:31.285001 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.284985    2575 server.go:1268] "Started kubelet"
Sep 08 15:17:31.285131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.285113    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 15:17:31.285485 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.285311    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 15:17:31.285608 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.285499    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 15:17:31.285608 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.285555    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 15:17:31.285608 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.285359    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 15:17:31.285732 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.285609    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 15:17:31.287134 ip-10-0-142-52 systemd[1]: Started Kubernetes Kubelet.
Sep 08 15:17:31.287507 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.287484    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 15:17:31.289773 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.289741    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f6c8"
Sep 08 15:17:31.290768 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.290748    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 15:17:31.292447 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.292404    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 15:17:31.293445 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.292407    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-52.ec2.internal.18d361797566444f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-52.ec2.internal,UID:ip-10-0-142-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-52.ec2.internal,},FirstTimestamp:2026-09-08 15:17:31.284792399 +0000 UTC m=+0.360084684,LastTimestamp:2026-09-08 15:17:31.284792399 +0000 UTC m=+0.360084684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-52.ec2.internal,}"
Sep 08 15:17:31.295103 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.295084    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 15:17:31.295455 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.295439    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 15:17:31.295614 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.295597    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.296205 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296185    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 15:17:31.296205 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296188    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 15:17:31.296331 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296214    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 15:17:31.296331 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296310    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 15:17:31.296331 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296316    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 15:17:31.296465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296451    2575 factory.go:55] Registering systemd factory
Sep 08 15:17:31.296494 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296470    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 08 15:17:31.296668 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296653    2575 factory.go:153] Registering CRI-O factory
Sep 08 15:17:31.296668 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296668    2575 factory.go:223] Registration of the crio container factory successfully
Sep 08 15:17:31.296750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296708    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 15:17:31.296750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296722    2575 factory.go:103] Registering Raw factory
Sep 08 15:17:31.296750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.296732    2575 manager.go:1196] Started watching for new ooms in manager
Sep 08 15:17:31.297249 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.297056    2575 manager.go:349] Starting recovery of all containers
Sep 08 15:17:31.302862 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.302828    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:31.306828 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.306815    2575 manager.go:354] Recovery completed
Sep 08 15:17:31.307435 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.307404    2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-52.ec2.internal\" not found" node="ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.308472 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.308454    2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 08 15:17:31.311523 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.311509    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.313284 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313270    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.313365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313300    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.313365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313319    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.313921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313909    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 15:17:31.313973 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313920    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 15:17:31.313973 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313936    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:17:31.314034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.313921    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.314651 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.314637    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.314706 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.314663    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.314706 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.314673    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.314940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.314929    2575 policy_none.go:49] "None policy: Start"
Sep 08 15:17:31.314982 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.314944    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 15:17:31.314982 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.314954    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 15:17:31.315900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.315889    2575 policy_none.go:47] "Start"
Sep 08 15:17:31.350417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350401    2575 manager.go:337] "Starting Device Plugin manager"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.350449    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350460    2575 server.go:87] "Starting device plugin registration server"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350686    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350700    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350858    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350924    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.350932    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.351325    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 15:17:31.365016 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.351375    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.433776 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.433756    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 15:17:31.435018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.435001    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 15:17:31.435077 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.435060    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 15:17:31.435111 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.435084    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 15:17:31.435142 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.435126    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 15:17:31.438291 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.438273    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:31.451593 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.451579    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.452306 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.452289    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.452391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.452320    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.452391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.452333    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.452391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.452353    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.459347 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.459245    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.459428 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.459356    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-52.ec2.internal\": node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.472175 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.472133    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.535896 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.535873    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"]
Sep 08 15:17:31.535980 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.535953    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.537288 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.537272    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.537353 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.537300    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.537353 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.537314    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.538528 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.538516    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.538666 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.538653    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.538703 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.538678    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.539241 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.539225    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.539241 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.539234    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.539402 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.539256    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.539402 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.539271    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.539402 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.539257    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.539402 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.539392    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.540358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.540340    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.540441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.540370    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:17:31.541029 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.541011    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:17:31.541113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.541045    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:17:31.541113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.541058    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:17:31.567379 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.567358    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-52.ec2.internal\" not found" node="ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.571486 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.571471    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-52.ec2.internal\" not found" node="ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.572471 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.572457    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.597744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.597722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.597809 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.597754    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.597809 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.597770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f119684184aeb2e23bdcf787e218d99-config\") pod \"kube-apiserver-proxy-ip-10-0-142-52.ec2.internal\" (UID: \"8f119684184aeb2e23bdcf787e218d99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.673268 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.673239    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.698672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.698648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.698771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.698679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.698771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.698704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f119684184aeb2e23bdcf787e218d99-config\") pod \"kube-apiserver-proxy-ip-10-0-142-52.ec2.internal\" (UID: \"8f119684184aeb2e23bdcf787e218d99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.698771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.698752    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.698771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.698762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f119684184aeb2e23bdcf787e218d99-config\") pod \"kube-apiserver-proxy-ip-10-0-142-52.ec2.internal\" (UID: \"8f119684184aeb2e23bdcf787e218d99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.698943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.698772    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.773961 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.773917    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.870566 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.870538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.874514 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.874245    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:31.874514 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:31.874291    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:31.974322 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:31.974292    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.074800 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:32.074747    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.175354 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:32.175333    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.192111 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.192081    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:32.232805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.232779    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 15:17:32.233197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.232960    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:17:32.233197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.232969    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:17:32.233197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.232975    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:17:32.276167 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:32.276145    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.295712 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.295667    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 15:12:31 +0000 UTC" deadline="2028-04-20 23:08:37.886681809 +0000 UTC"
Sep 08 15:17:32.295712 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.295704    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14167h51m5.590980329s"
Sep 08 15:17:32.376771 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:32.376702    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.477234 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:32.477212    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.577710 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:32.577682    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 08 15:17:32.636536 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.636489    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:32.654436 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:32.654413    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f119684184aeb2e23bdcf787e218d99.slice/crio-c860b52236d0dff712020a5baa64ed027f1cd39b876ac2d5eab3572cd11a1aef WatchSource:0}: Error finding container c860b52236d0dff712020a5baa64ed027f1cd39b876ac2d5eab3572cd11a1aef: Status 404 returned error can't find the container with id c860b52236d0dff712020a5baa64ed027f1cd39b876ac2d5eab3572cd11a1aef
Sep 08 15:17:32.659137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.659122    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:17:32.695956 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.695933    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:32.707683 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.707667    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:17:32.708717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.708706    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 08 15:17:32.718959 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:32.718945    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:17:33.075653 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.075629    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:33.275314 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.275258    2575 apiserver.go:52] "Watching apiserver"
Sep 08 15:17:33.283877 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.283859    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 15:17:33.285047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.284581    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-kmlj7","openshift-network-diagnostics/network-check-target-mbkb4","openshift-network-operator/iptables-alerter-gh4d8","kube-system/global-pull-secret-syncer-m66rl","kube-system/konnectivity-agent-fmx62","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg","openshift-cluster-node-tuning-operator/tuned-zshc7","openshift-image-registry/node-ca-gzb7q","openshift-ovn-kubernetes/ovnkube-node-l8264","kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal","openshift-dns/node-resolver-bc6nc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal","openshift-multus/multus-additional-cni-plugins-jgclk","openshift-multus/multus-k92tr"]
Sep 08 15:17:33.287023 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.287006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.288947 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.288928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.289630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.289614    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.289795 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.289783    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7nkfn\""
Sep 08 15:17:33.289987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.289976    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.291139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.291125    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.291377 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.291361    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 15:17:33.291479 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.291462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.291891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.291876    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.291970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.291934    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 15:17:33.292063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.292049    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 15:17:33.292129 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.292087    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5vf69\""
Sep 08 15:17:33.293458 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.293436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.293775 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.293521    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:33.293775 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.293601    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.293907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.293778    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.293996 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.293979    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-85dt6\""
Sep 08 15:17:33.294047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.294000    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 15:17:33.295421 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.295405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.295768 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.295752    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 15:17:33.297874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.297601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.297874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.297672    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 15:17:33.297874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.297684    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 15:17:33.297874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.297771    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x5ww6\""
Sep 08 15:17:33.299872 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.299854    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 15:17:33.299985 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.299911    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.300042 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.300031    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c6hlm\""
Sep 08 15:17:33.300295 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.300279    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.300418 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.300405    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.302002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.301989    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:33.302069 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.302049    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:33.302578 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.302563    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.304060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.303940    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 15:17:33.304060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.303950    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 15:17:33.304060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.303957    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.304060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.303947    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 15:17:33.304060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.303962    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 15:17:33.304060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.304045    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mzqpl\""
Sep 08 15:17:33.304352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.304338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:33.304401 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.304388    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:33.306155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ac627f-21b3-469b-b8ed-478d372b7f0a-dbus\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.306242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.306242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cnibin\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.306242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.306358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-socket-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.306358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-modprobe-d\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6371911b-a1b5-4a63-a69f-cfdda1648782-konnectivity-ca\") pod \"konnectivity-agent-fmx62\" (UID: \"6371911b-a1b5-4a63-a69f-cfdda1648782\") " pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.306358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-host\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306488 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19261b42-37ac-452c-828f-d562f12460f4-tmp\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306488 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbh8h\" (UniqueName: \"kubernetes.io/projected/72e72db5-7202-4b58-8ac2-c378206aa2c7-kube-api-access-bbh8h\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.306554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.306554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-kubernetes\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysctl-d\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306581    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.306641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9n9\" (UniqueName: \"kubernetes.io/projected/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-kube-api-access-ld9n9\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.306745 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-sys\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306745 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.306820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-registration-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.306820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-sys-fs\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.306820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-run\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306948 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19261b42-37ac-452c-828f-d562f12460f4-etc-tuned\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.306948 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.306948 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.306948 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ac627f-21b3-469b-b8ed-478d372b7f0a-kubelet-config\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.307074 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.306956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysconfig\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.307074 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-lib-modules\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.307164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-var-lib-kubelet\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.307164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvl9\" (UniqueName: \"kubernetes.io/projected/19261b42-37ac-452c-828f-d562f12460f4-kube-api-access-2tvl9\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.307255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.307255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6371911b-a1b5-4a63-a69f-cfdda1648782-agent-certs\") pod \"konnectivity-agent-fmx62\" (UID: \"6371911b-a1b5-4a63-a69f-cfdda1648782\") " pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.307255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysctl-conf\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.307387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czv4p\" (UniqueName: \"kubernetes.io/projected/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-kube-api-access-czv4p\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.307387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72e72db5-7202-4b58-8ac2-c378206aa2c7-host-slash\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.307387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-device-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.307387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.307387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307376    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:17:33.307573 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-systemd\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.307573 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.307413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72e72db5-7202-4b58-8ac2-c378206aa2c7-iptables-alerter-script\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.308395 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.308377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.309007 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.308989    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67dtq\""
Sep 08 15:17:33.309134 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.309119    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 15:17:33.310626 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.310611    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.310704 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.310690    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.311080 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.311064    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.311141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.311126    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g4nlv\""
Sep 08 15:17:33.313544 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.313339    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 15:17:33.313979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.313958    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 15:17:33.314065 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.314046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 15:17:33.314600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.314585    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z55zc\""
Sep 08 15:17:33.326510 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.326495    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kv4pq"
Sep 08 15:17:33.332433 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.332420    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kv4pq"
Sep 08 15:17:33.397645 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.397620    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 15:17:33.408000 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.407982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-sys-fs\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.408093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.408093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rq22\" (UniqueName: \"kubernetes.io/projected/c094bb8e-9ba5-40ad-97c5-3d236e59f365-kube-api-access-2rq22\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.408093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eff82d57-cf61-4995-8dd9-d3da3aa6269d-host\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.408201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.408201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-sys-fs\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.408201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9bdc\" (UniqueName: \"kubernetes.io/projected/4317d79b-6d07-4211-8ecc-3d56b8d5123a-kube-api-access-k9bdc\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:33.408201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-cni-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-cnibin\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvl9\" (UniqueName: \"kubernetes.io/projected/19261b42-37ac-452c-828f-d562f12460f4-kube-api-access-2tvl9\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdn4\" (UniqueName: \"kubernetes.io/projected/eff82d57-cf61-4995-8dd9-d3da3aa6269d-kube-api-access-4rdn4\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-netns\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-cni-bin\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408329    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-kubelet\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-kubelet\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlflm\" (UniqueName: \"kubernetes.io/projected/c43a465a-b81d-48c9-93c6-ac212051f4f0-kube-api-access-vlflm\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-device-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-device-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72e72db5-7202-4b58-8ac2-c378206aa2c7-iptables-alerter-script\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ac627f-21b3-469b-b8ed-478d372b7f0a-dbus\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-log-socket\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408577    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408606    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-conf-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cnibin\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c094bb8e-9ba5-40ad-97c5-3d236e59f365-hosts-file\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-var-lib-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cnibin\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ac627f-21b3-469b-b8ed-478d372b7f0a-dbus\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:33.408760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-socket-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-modprobe-d\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6371911b-a1b5-4a63-a69f-cfdda1648782-konnectivity-ca\") pod \"konnectivity-agent-fmx62\" (UID: \"6371911b-a1b5-4a63-a69f-cfdda1648782\") " pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-ovn\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408890    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-modprobe-d\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-socket-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-cni-netd\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c43a465a-b81d-48c9-93c6-ac212051f4f0-cni-binary-copy\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-socket-dir-parent\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.408994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-host\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbh8h\" (UniqueName: \"kubernetes.io/projected/72e72db5-7202-4b58-8ac2-c378206aa2c7-kube-api-access-bbh8h\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-cni-multus\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-cni-bin\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovnkube-config\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-env-overrides\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovn-node-metrics-cert\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.409446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-run-netns\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-node-log\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-registration-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-host\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-run\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409255    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-registration-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19261b42-37ac-452c-828f-d562f12460f4-etc-tuned\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-run\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ac627f-21b3-469b-b8ed-478d372b7f0a-kubelet-config\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6371911b-a1b5-4a63-a69f-cfdda1648782-agent-certs\") pod \"konnectivity-agent-fmx62\" (UID: \"6371911b-a1b5-4a63-a69f-cfdda1648782\") " pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysconfig\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ac627f-21b3-469b-b8ed-478d372b7f0a-kubelet-config\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysconfig\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-lib-modules\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-var-lib-kubelet\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72e72db5-7202-4b58-8ac2-c378206aa2c7-iptables-alerter-script\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.410139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-var-lib-kubelet\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-slash\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-lib-modules\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovnkube-script-lib\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-hostroot\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409662    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-daemon-config\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysctl-conf\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czv4p\" (UniqueName: \"kubernetes.io/projected/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-kube-api-access-czv4p\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409799    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6371911b-a1b5-4a63-a69f-cfdda1648782-konnectivity-ca\") pod \"konnectivity-agent-fmx62\" (UID: \"6371911b-a1b5-4a63-a69f-cfdda1648782\") " pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72e72db5-7202-4b58-8ac2-c378206aa2c7-host-slash\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409897    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-system-cni-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-systemd\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysctl-conf\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eff82d57-cf61-4995-8dd9-d3da3aa6269d-serviceca\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72e72db5-7202-4b58-8ac2-c378206aa2c7-host-slash\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-multus-certs\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.410831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409963    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-systemd\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.409994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-etc-kubernetes\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-k8s-cni-cncf-io\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c094bb8e-9ba5-40ad-97c5-3d236e59f365-tmp-dir\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-systemd\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19261b42-37ac-452c-828f-d562f12460f4-tmp\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-systemd-units\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmc8z\" (UniqueName: \"kubernetes.io/projected/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-kube-api-access-bmc8z\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-kubernetes\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.410291    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysctl-d\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-kubernetes\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9n9\" (UniqueName: \"kubernetes.io/projected/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-kube-api-access-ld9n9\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.411424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-etc-sysctl-d\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.410442    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:33.910406451 +0000 UTC m=+2.985698725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-sys\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19261b42-37ac-452c-828f-d562f12460f4-sys\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-etc-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.410715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.411970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.411160    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.412723 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.412703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19261b42-37ac-452c-828f-d562f12460f4-etc-tuned\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.412793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.412737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19261b42-37ac-452c-828f-d562f12460f4-tmp\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.412954 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.412938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6371911b-a1b5-4a63-a69f-cfdda1648782-agent-certs\") pod \"konnectivity-agent-fmx62\" (UID: \"6371911b-a1b5-4a63-a69f-cfdda1648782\") " pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.418927 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.418883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvl9\" (UniqueName: \"kubernetes.io/projected/19261b42-37ac-452c-828f-d562f12460f4-kube-api-access-2tvl9\") pod \"tuned-zshc7\" (UID: \"19261b42-37ac-452c-828f-d562f12460f4\") " pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.421036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.421014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9n9\" (UniqueName: \"kubernetes.io/projected/ed22bc20-2c4d-4c50-a0b3-ace51bdd9328-kube-api-access-ld9n9\") pod \"aws-ebs-csi-driver-node-jbpgg\" (UID: \"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.421123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.421042    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbh8h\" (UniqueName: \"kubernetes.io/projected/72e72db5-7202-4b58-8ac2-c378206aa2c7-kube-api-access-bbh8h\") pod \"iptables-alerter-gh4d8\" (UID: \"72e72db5-7202-4b58-8ac2-c378206aa2c7\") " pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.421123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.421105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czv4p\" (UniqueName: \"kubernetes.io/projected/abd18e81-3dc2-4b04-b48e-f0ba03f796d1-kube-api-access-czv4p\") pod \"multus-additional-cni-plugins-jgclk\" (UID: \"abd18e81-3dc2-4b04-b48e-f0ba03f796d1\") " pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.438557 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.438522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal" event={"ID":"8f119684184aeb2e23bdcf787e218d99","Type":"ContainerStarted","Data":"c860b52236d0dff712020a5baa64ed027f1cd39b876ac2d5eab3572cd11a1aef"}
Sep 08 15:17:33.469692 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.469675    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:33.511385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-etc-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rq22\" (UniqueName: \"kubernetes.io/projected/c094bb8e-9ba5-40ad-97c5-3d236e59f365-kube-api-access-2rq22\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eff82d57-cf61-4995-8dd9-d3da3aa6269d-host\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9bdc\" (UniqueName: \"kubernetes.io/projected/4317d79b-6d07-4211-8ecc-3d56b8d5123a-kube-api-access-k9bdc\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-cni-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-cnibin\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511468 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511452    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-etc-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eff82d57-cf61-4995-8dd9-d3da3aa6269d-host\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdn4\" (UniqueName: \"kubernetes.io/projected/eff82d57-cf61-4995-8dd9-d3da3aa6269d-kube-api-access-4rdn4\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-cni-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-netns\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-cnibin\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-netns\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-cni-bin\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-kubelet\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-kubelet\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511612    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-cni-bin\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511644    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-kubelet\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-kubelet\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlflm\" (UniqueName: \"kubernetes.io/projected/c43a465a-b81d-48c9-93c6-ac212051f4f0-kube-api-access-vlflm\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-log-socket\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-conf-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.511749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c094bb8e-9ba5-40ad-97c5-3d236e59f365-hosts-file\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-var-lib-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511756    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-conf-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c094bb8e-9ba5-40ad-97c5-3d236e59f365-hosts-file\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-log-socket\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511807    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-ovn\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511838    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-ovn\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-cni-netd\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-var-lib-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.511885    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c43a465a-b81d-48c9-93c6-ac212051f4f0-cni-binary-copy\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-socket-dir-parent\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-cni-multus\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-cni-netd\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.512515 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.511951    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:34.011935836 +0000 UTC m=+3.087228104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-var-lib-cni-multus\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-cni-bin\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-socket-dir-parent\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.511996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovnkube-config\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-env-overrides\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovn-node-metrics-cert\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-run-netns\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-node-log\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-cni-bin\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-run-netns\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-node-log\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-slash\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovnkube-script-lib\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-openvswitch\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-host-slash\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-hostroot\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-daemon-config\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-system-cni-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eff82d57-cf61-4995-8dd9-d3da3aa6269d-serviceca\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-multus-certs\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-etc-kubernetes\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-env-overrides\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-k8s-cni-cncf-io\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-multus-certs\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512403    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-system-cni-dir\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c094bb8e-9ba5-40ad-97c5-3d236e59f365-tmp-dir\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-systemd\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovnkube-config\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-systemd-units\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-hostroot\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmc8z\" (UniqueName: \"kubernetes.io/projected/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-kube-api-access-bmc8z\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-host-run-k8s-cni-cncf-io\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-run-systemd\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c43a465a-b81d-48c9-93c6-ac212051f4f0-etc-kubernetes\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.513631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512679    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovnkube-script-lib\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.514155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-systemd-units\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.514155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c094bb8e-9ba5-40ad-97c5-3d236e59f365-tmp-dir\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.514155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c43a465a-b81d-48c9-93c6-ac212051f4f0-multus-daemon-config\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.514155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512830    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eff82d57-cf61-4995-8dd9-d3da3aa6269d-serviceca\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.514155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.512966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c43a465a-b81d-48c9-93c6-ac212051f4f0-cni-binary-copy\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.514155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.513977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-ovn-node-metrics-cert\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.520533 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.520515    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:17:33.520629 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.520537    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:17:33.520629 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.520549    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:33.520629 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.520611    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:17:34.02059598 +0000 UTC m=+3.095888263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:33.521683 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.521660    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rq22\" (UniqueName: \"kubernetes.io/projected/c094bb8e-9ba5-40ad-97c5-3d236e59f365-kube-api-access-2rq22\") pod \"node-resolver-bc6nc\" (UID: \"c094bb8e-9ba5-40ad-97c5-3d236e59f365\") " pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.522201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.522184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9bdc\" (UniqueName: \"kubernetes.io/projected/4317d79b-6d07-4211-8ecc-3d56b8d5123a-kube-api-access-k9bdc\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:33.522275 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.522258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlflm\" (UniqueName: \"kubernetes.io/projected/c43a465a-b81d-48c9-93c6-ac212051f4f0-kube-api-access-vlflm\") pod \"multus-k92tr\" (UID: \"c43a465a-b81d-48c9-93c6-ac212051f4f0\") " pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.522433 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.522420    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdn4\" (UniqueName: \"kubernetes.io/projected/eff82d57-cf61-4995-8dd9-d3da3aa6269d-kube-api-access-4rdn4\") pod \"node-ca-gzb7q\" (UID: \"eff82d57-cf61-4995-8dd9-d3da3aa6269d\") " pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.522925 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.522912    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmc8z\" (UniqueName: \"kubernetes.io/projected/ea8a1452-c2ec-449d-abf3-818d7fffb9d4-kube-api-access-bmc8z\") pod \"ovnkube-node-l8264\" (UID: \"ea8a1452-c2ec-449d-abf3-818d7fffb9d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.595746 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.595707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zshc7"
Sep 08 15:17:33.601335 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.601319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jgclk"
Sep 08 15:17:33.604145 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.604126    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19261b42_37ac_452c_828f_d562f12460f4.slice/crio-b216dee7b2323aa14af28c38d3823e329ed590cb2cc233636cdd49b87d95b61c WatchSource:0}: Error finding container b216dee7b2323aa14af28c38d3823e329ed590cb2cc233636cdd49b87d95b61c: Status 404 returned error can't find the container with id b216dee7b2323aa14af28c38d3823e329ed590cb2cc233636cdd49b87d95b61c
Sep 08 15:17:33.608556 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.608534    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd18e81_3dc2_4b04_b48e_f0ba03f796d1.slice/crio-6e4055aa70351d3d1d7f2dd4af2fae203c038301f975abaa613180daf825db83 WatchSource:0}: Error finding container 6e4055aa70351d3d1d7f2dd4af2fae203c038301f975abaa613180daf825db83: Status 404 returned error can't find the container with id 6e4055aa70351d3d1d7f2dd4af2fae203c038301f975abaa613180daf825db83
Sep 08 15:17:33.609176 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.609165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gh4d8"
Sep 08 15:17:33.615026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.615012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:33.617171 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.617153    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e72db5_7202_4b58_8ac2_c378206aa2c7.slice/crio-86583325167ddbafed3a67c2fad931c7256d9e326480f6ff10c8c51ecb5f9bf1 WatchSource:0}: Error finding container 86583325167ddbafed3a67c2fad931c7256d9e326480f6ff10c8c51ecb5f9bf1: Status 404 returned error can't find the container with id 86583325167ddbafed3a67c2fad931c7256d9e326480f6ff10c8c51ecb5f9bf1
Sep 08 15:17:33.620742 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.620725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg"
Sep 08 15:17:33.621920 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.621902    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6371911b_a1b5_4a63_a69f_cfdda1648782.slice/crio-070e6790afa8e75e8fa97b26778e70c802bb5f9a7459171bda90e85da1c704a9 WatchSource:0}: Error finding container 070e6790afa8e75e8fa97b26778e70c802bb5f9a7459171bda90e85da1c704a9: Status 404 returned error can't find the container with id 070e6790afa8e75e8fa97b26778e70c802bb5f9a7459171bda90e85da1c704a9
Sep 08 15:17:33.624652 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.624636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:33.628398 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.628383    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded22bc20_2c4d_4c50_a0b3_ace51bdd9328.slice/crio-c9cdf55e1e83ebd55e73641130f87fe96d132f330ff742c1915921c46c302b18 WatchSource:0}: Error finding container c9cdf55e1e83ebd55e73641130f87fe96d132f330ff742c1915921c46c302b18: Status 404 returned error can't find the container with id c9cdf55e1e83ebd55e73641130f87fe96d132f330ff742c1915921c46c302b18
Sep 08 15:17:33.629131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.629115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k92tr"
Sep 08 15:17:33.632781 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.632763    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8a1452_c2ec_449d_abf3_818d7fffb9d4.slice/crio-8c0849a5e7d6bcaa2cb81295812c36aae31b2970aa20ed25b37471c2da8a72c4 WatchSource:0}: Error finding container 8c0849a5e7d6bcaa2cb81295812c36aae31b2970aa20ed25b37471c2da8a72c4: Status 404 returned error can't find the container with id 8c0849a5e7d6bcaa2cb81295812c36aae31b2970aa20ed25b37471c2da8a72c4
Sep 08 15:17:33.633981 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.633963    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bc6nc"
Sep 08 15:17:33.637268 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.637253    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43a465a_b81d_48c9_93c6_ac212051f4f0.slice/crio-ac77a56047f5f724eaff6cc78f9f1d31266f8772a32437a5fa9565e8f8137a29 WatchSource:0}: Error finding container ac77a56047f5f724eaff6cc78f9f1d31266f8772a32437a5fa9565e8f8137a29: Status 404 returned error can't find the container with id ac77a56047f5f724eaff6cc78f9f1d31266f8772a32437a5fa9565e8f8137a29
Sep 08 15:17:33.638312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.638296    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gzb7q"
Sep 08 15:17:33.641723 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.641710    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc094bb8e_9ba5_40ad_97c5_3d236e59f365.slice/crio-b076b7bf91da507aefa6ffe6ce2d902048dac40195e014ed2bdbbe4190c8d0eb WatchSource:0}: Error finding container b076b7bf91da507aefa6ffe6ce2d902048dac40195e014ed2bdbbe4190c8d0eb: Status 404 returned error can't find the container with id b076b7bf91da507aefa6ffe6ce2d902048dac40195e014ed2bdbbe4190c8d0eb
Sep 08 15:17:33.648041 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:17:33.648022    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff82d57_cf61_4995_8dd9_d3da3aa6269d.slice/crio-cfe9b662c1adbde65bdb3da09d034280e55831f56b6e16e05a466963ed97b8b6 WatchSource:0}: Error finding container cfe9b662c1adbde65bdb3da09d034280e55831f56b6e16e05a466963ed97b8b6: Status 404 returned error can't find the container with id cfe9b662c1adbde65bdb3da09d034280e55831f56b6e16e05a466963ed97b8b6
Sep 08 15:17:33.914367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:33.914308    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:33.914489 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.914438    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:33.914489 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:33.914485    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:34.914473114 +0000 UTC m=+3.989765383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:34.015102 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.015063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:34.015249 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.015182    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:34.015249 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.015227    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:35.015215464 +0000 UTC m=+4.090507744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:34.116172 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.116138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:34.116308 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.116263    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:17:34.116308 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.116299    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:17:34.116308 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.116309    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:34.116431 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.116347    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:17:35.116336358 +0000 UTC m=+4.191628628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:34.335217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.335132    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:12:33 +0000 UTC" deadline="2028-05-17 17:45:03.245546736 +0000 UTC"
Sep 08 15:17:34.335217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.335176    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14810h27m28.910374797s"
Sep 08 15:17:34.435718 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.435697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:34.435902 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.435796    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:34.442927 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.442863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gh4d8" event={"ID":"72e72db5-7202-4b58-8ac2-c378206aa2c7","Type":"ContainerStarted","Data":"86583325167ddbafed3a67c2fad931c7256d9e326480f6ff10c8c51ecb5f9bf1"}
Sep 08 15:17:34.444904 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.444883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerStarted","Data":"6e4055aa70351d3d1d7f2dd4af2fae203c038301f975abaa613180daf825db83"}
Sep 08 15:17:34.446348 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.446311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg" event={"ID":"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328","Type":"ContainerStarted","Data":"c9cdf55e1e83ebd55e73641130f87fe96d132f330ff742c1915921c46c302b18"}
Sep 08 15:17:34.447565 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.447535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmx62" event={"ID":"6371911b-a1b5-4a63-a69f-cfdda1648782","Type":"ContainerStarted","Data":"070e6790afa8e75e8fa97b26778e70c802bb5f9a7459171bda90e85da1c704a9"}
Sep 08 15:17:34.448856 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.448786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zshc7" event={"ID":"19261b42-37ac-452c-828f-d562f12460f4","Type":"ContainerStarted","Data":"b216dee7b2323aa14af28c38d3823e329ed590cb2cc233636cdd49b87d95b61c"}
Sep 08 15:17:34.451383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.451355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gzb7q" event={"ID":"eff82d57-cf61-4995-8dd9-d3da3aa6269d","Type":"ContainerStarted","Data":"cfe9b662c1adbde65bdb3da09d034280e55831f56b6e16e05a466963ed97b8b6"}
Sep 08 15:17:34.452916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.452894    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bc6nc" event={"ID":"c094bb8e-9ba5-40ad-97c5-3d236e59f365","Type":"ContainerStarted","Data":"b076b7bf91da507aefa6ffe6ce2d902048dac40195e014ed2bdbbe4190c8d0eb"}
Sep 08 15:17:34.454466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.454443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k92tr" event={"ID":"c43a465a-b81d-48c9-93c6-ac212051f4f0","Type":"ContainerStarted","Data":"ac77a56047f5f724eaff6cc78f9f1d31266f8772a32437a5fa9565e8f8137a29"}
Sep 08 15:17:34.455920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.455899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"8c0849a5e7d6bcaa2cb81295812c36aae31b2970aa20ed25b37471c2da8a72c4"}
Sep 08 15:17:34.923822 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:34.923252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:34.923822 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.923434    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:34.923822 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:34.923491    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:36.923473789 +0000 UTC m=+5.998766074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:35.024803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.024267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:35.024803 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.024405    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:35.024803 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.024461    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:37.024442663 +0000 UTC m=+6.099734955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:35.124898 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.124864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:35.125069 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.125047    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:17:35.125069 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.125068    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:17:35.125173 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.125083    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:35.125173 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.125134    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:17:37.125116869 +0000 UTC m=+6.200409142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:35.276206 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.276180    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:17:35.335331 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.335275    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:12:33 +0000 UTC" deadline="2028-03-06 19:36:16.290800923 +0000 UTC"
Sep 08 15:17:35.335331 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.335331    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13084h18m40.955474013s"
Sep 08 15:17:35.437231 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.437206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:35.437358 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.437328    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:35.437721 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.437701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:35.437819 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:35.437799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:35.460518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.460486    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" event={"ID":"7e4f9d586d66128144e4e5ad291a5b68","Type":"ContainerStarted","Data":"43b0e7912b7fa4b98414ba7c3ba2ef4725539ea4b4b381c3596563542f69eab5"}
Sep 08 15:17:35.466362 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:35.466336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal" event={"ID":"8f119684184aeb2e23bdcf787e218d99","Type":"ContainerStarted","Data":"9debd2c11f8accfd7c7ff46937cf753a29a20b8156db645429ff2ad4ffe4feb5"}
Sep 08 15:17:36.436078 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:36.435864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:36.436465 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:36.436172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:36.940009 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:36.939943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:36.940198 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:36.940081    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:36.940198 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:36.940142    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:40.940123883 +0000 UTC m=+10.015416168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:37.040764 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:37.040722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:37.040966 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.040904    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:37.040966 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.040966    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:41.040948297 +0000 UTC m=+10.116240571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:37.141697 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:37.141662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:37.141905 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.141885    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:17:37.141905 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.141904    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:17:37.142023 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.141917    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:37.142023 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.141975    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:17:41.141955795 +0000 UTC m=+10.217248080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:37.438178 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:37.438105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:37.438626 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.438279    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:37.438686 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:37.438647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:37.438822 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:37.438799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:38.435875 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:38.435827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:38.436054 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:38.435976    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:39.435962 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:39.435932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:39.436408 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:39.436107    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:39.436408 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:39.436149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:39.436408 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:39.436234    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:40.435441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:40.435406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:40.435631 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:40.435583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:40.972034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:40.971996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:40.972454 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:40.972194    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:40.972454 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:40.972254    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:48.972235812 +0000 UTC m=+18.047528097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:41.073167 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:41.073135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:41.073338 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.073277    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:41.073338 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.073330    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:17:49.073315676 +0000 UTC m=+18.148607962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:41.173988 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:41.173953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:41.174162 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.174122    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:17:41.174162 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.174136    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:17:41.174162 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.174149    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:41.174308 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.174194    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:17:49.174181803 +0000 UTC m=+18.249474087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:41.436889 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:41.436832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:41.437065 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.436970    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:41.437387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:41.437371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:41.437480 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:41.437465    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:42.435495 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:42.435463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:42.435979 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:42.435596    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:43.436967 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:43.436934    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:43.437411 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:43.436939    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:43.437411 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:43.437078    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:43.437411 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:43.437124    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:44.435726 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:44.435693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:44.435924 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:44.435815    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:45.435737 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:45.435708    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:45.436190 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:45.435827    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:45.436190 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:45.435882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:45.436190 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:45.436001    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:46.435775 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:46.435741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:46.436213 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:46.435867    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:47.435914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:47.435885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:47.436332 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:47.435923    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:47.436332 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:47.435983    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:47.436332 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:47.436052    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:48.435993 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:48.435963    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:48.436436 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:48.436072    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:49.031701 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:49.031669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:49.031995 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.031809    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:49.031995 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.031892    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:18:05.031876542 +0000 UTC m=+34.107168813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:17:49.132251 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:49.132217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:49.132393 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.132346    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:49.132443 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.132413    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:18:05.132391268 +0000 UTC m=+34.207683546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:17:49.233030 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:49.232988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:49.233198 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.233153    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:17:49.233198 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.233174    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:17:49.233198 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.233187    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:49.233345 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.233247    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:05.233229355 +0000 UTC m=+34.308521648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:17:49.435585 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:49.435557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:49.435585 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:49.435569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:49.435817 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.435685    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:49.435897 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:49.435817    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:50.436099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:50.436061    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:50.436482 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:50.436189    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:51.435831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:51.435793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:51.436018 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:51.435906    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:51.436104 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:51.436041    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:51.436177 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:51.436154    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:52.435406 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.435245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:52.435508 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:52.435483    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:52.505597 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.505573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zshc7" event={"ID":"19261b42-37ac-452c-828f-d562f12460f4","Type":"ContainerStarted","Data":"cf432deeadce8c76d78bb066c1d7f87f4762384035ff28c442e6dab966ac47c4"}
Sep 08 15:17:52.507453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.507423    2575 generic.go:356] "Generic (PLEG): container finished" podID="7e4f9d586d66128144e4e5ad291a5b68" containerID="9efdfe213dede77d76c881ede47e70f9e92b5024c16faa6f667735bf930fc9d7" exitCode=0
Sep 08 15:17:52.507941 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.507914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" event={"ID":"7e4f9d586d66128144e4e5ad291a5b68","Type":"ContainerDied","Data":"9efdfe213dede77d76c881ede47e70f9e92b5024c16faa6f667735bf930fc9d7"}
Sep 08 15:17:52.509413 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.509395    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gzb7q" event={"ID":"eff82d57-cf61-4995-8dd9-d3da3aa6269d","Type":"ContainerStarted","Data":"60c37826ec8bd9b8d3d63ed0700f682a4c2bcc0514d045049aa3ddde808f10ad"}
Sep 08 15:17:52.511155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.511132    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bc6nc" event={"ID":"c094bb8e-9ba5-40ad-97c5-3d236e59f365","Type":"ContainerStarted","Data":"e409011a696d1a9527a5f5b3542de56daa0de2c8d6036800b379d23e47d3ae65"}
Sep 08 15:17:52.512883 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.512837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k92tr" event={"ID":"c43a465a-b81d-48c9-93c6-ac212051f4f0","Type":"ContainerStarted","Data":"30d098c4b8cfaeac40fdead4cae19723947b53fda18326d390f8ec796a9bd95f"}
Sep 08 15:17:52.520056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.518337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"91151729194a982445d4d7c24611d572c52586c9da0ddc766c848981378921d0"}
Sep 08 15:17:52.520056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.518410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"4f45a5884aefa8eb865bb004fe20f9007447a6a903bf3b9fa8993d20994124f6"}
Sep 08 15:17:52.520056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.518464    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"9d626234016b79257c97ad509832e0baea3f934e96c53a35c229bdaeea6233eb"}
Sep 08 15:17:52.520056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.518514    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"c1e6020bbf444dc9e8476212acc26293e572605fa6b343a27f062a768d9cbd12"}
Sep 08 15:17:52.520056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.518561    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"8f64debce90cb2d5566578354d60d62bb087f81c2b8dae3ee6f5756f1243a1f2"}
Sep 08 15:17:52.521763 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.521743    2575 generic.go:356] "Generic (PLEG): container finished" podID="abd18e81-3dc2-4b04-b48e-f0ba03f796d1" containerID="12270d7b892a2c65e10767ccabf1bc80c58747cf8500acbcf9f21e7eb8a11c49" exitCode=0
Sep 08 15:17:52.521889 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.521826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerDied","Data":"12270d7b892a2c65e10767ccabf1bc80c58747cf8500acbcf9f21e7eb8a11c49"}
Sep 08 15:17:52.523601 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.523579    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg" event={"ID":"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328","Type":"ContainerStarted","Data":"8014b12e176d3cf7c382855d649f7963b24f4bd0c91302e0ae6b448f62b9ccbd"}
Sep 08 15:17:52.525367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.525343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmx62" event={"ID":"6371911b-a1b5-4a63-a69f-cfdda1648782","Type":"ContainerStarted","Data":"e83fd54f667c61e854b2acd1c910771b80b87544c15074cf1c83a7ae27d24f99"}
Sep 08 15:17:52.534160 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.534138    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:52.535011 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.534993    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:52.535132 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.535085    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal" podStartSLOduration=20.535068086 podStartE2EDuration="20.535068086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:35.487052848 +0000 UTC m=+4.562345157" watchObservedRunningTime="2026-09-08 15:17:52.535068086 +0000 UTC m=+21.610360381"
Sep 08 15:17:52.535753 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.535721    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zshc7" podStartSLOduration=3.218742599 podStartE2EDuration="21.535710803s" totalImagesPullingTime="18.316968204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:52.534704501 +0000 UTC m=+21.609996794" watchObservedRunningTime="2026-09-08 15:17:52.535710803 +0000 UTC m=+21.611003096"
Sep 08 15:17:52.583183 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.583150    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bc6nc" podStartSLOduration=3.438381218 podStartE2EDuration="21.583136847s" totalImagesPullingTime="18.144755629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:52.58286505 +0000 UTC m=+21.658157353" watchObservedRunningTime="2026-09-08 15:17:52.583136847 +0000 UTC m=+21.658429137"
Sep 08 15:17:52.658438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.658404    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gzb7q" podStartSLOduration=3.519314573 podStartE2EDuration="21.65839269s" totalImagesPullingTime="18.139078117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:52.626261665 +0000 UTC m=+21.701553957" watchObservedRunningTime="2026-09-08 15:17:52.65839269 +0000 UTC m=+21.733684983"
Sep 08 15:17:52.739795 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.739605    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k92tr" podStartSLOduration=3.428573129 podStartE2EDuration="21.739589768s" totalImagesPullingTime="18.311016639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:52.737265648 +0000 UTC m=+21.812557940" watchObservedRunningTime="2026-09-08 15:17:52.739589768 +0000 UTC m=+21.814882061"
Sep 08 15:17:52.740101 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:52.740038    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fmx62" podStartSLOduration=3.608497313 podStartE2EDuration="21.740026677s" totalImagesPullingTime="18.131529364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:52.691936434 +0000 UTC m=+21.767228743" watchObservedRunningTime="2026-09-08 15:17:52.740026677 +0000 UTC m=+21.815318966"
Sep 08 15:17:53.435551 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.435477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:53.435696 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:53.435605    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:53.435696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.435663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:53.435806 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:53.435777    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:53.530770 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.530733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"bdce7bdcf8ed28d324f5e003d01512d0407c44ef676d53096ffda259c0183249"}
Sep 08 15:17:53.532546 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.532522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gh4d8" event={"ID":"72e72db5-7202-4b58-8ac2-c378206aa2c7","Type":"ContainerStarted","Data":"1a6328129e1ef0eb13e91c71885fdbdcbb92e8d15b800b910a9237687485e8b5"}
Sep 08 15:17:53.534219 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.534186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" event={"ID":"7e4f9d586d66128144e4e5ad291a5b68","Type":"ContainerStarted","Data":"2e07b646ec04a13c0e6b77f9f5346017ae1088279178dce7484fe3a844c5e9e7"}
Sep 08 15:17:53.603452 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.603405    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" podStartSLOduration=21.6033871 podStartE2EDuration="21.6033871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:53.602868621 +0000 UTC m=+22.678160933" watchObservedRunningTime="2026-09-08 15:17:53.6033871 +0000 UTC m=+22.678679396"
Sep 08 15:17:53.603943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:53.603904    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gh4d8" podStartSLOduration=4.300428748 podStartE2EDuration="22.603888985s" totalImagesPullingTime="18.303460237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:53.564188314 +0000 UTC m=+22.639480607" watchObservedRunningTime="2026-09-08 15:17:53.603888985 +0000 UTC m=+22.679181279"
Sep 08 15:17:54.436303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:54.435865    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:54.436303 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:54.435981    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:54.525421 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:54.525389    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 15:17:54.537525 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:54.537502    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:17:54.537887 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:54.537560    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg" event={"ID":"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328","Type":"ContainerStarted","Data":"8e3001e902fd55533c2c8e05b124ffab838bf2884fbb3e226efc2322cedc00a4"}
Sep 08 15:17:55.359501 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.359363    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T15:17:54.525410348Z","UUID":"12e425e9-9e40-4ad8-a8cb-96a29647e408","Handler":null,"Name":"","Endpoint":""}
Sep 08 15:17:55.360763 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.360743    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 15:17:55.360763 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.360774    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 15:17:55.435524 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.435493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:55.435653 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:55.435615    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:55.435696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.435672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:55.435774 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:55.435757    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:55.543815 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.543783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"832c4df8e6c8f69d35eecb61a28ac8bd5636e5f21f34d73d369b162ca1fa50cd"}
Sep 08 15:17:55.764516 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.764483    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:55.764703 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.764630    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:17:55.765108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:55.765088    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fmx62"
Sep 08 15:17:56.435912 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:56.435884    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:56.436096 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:56.435994    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:57.436384 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.436165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:57.436823 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.436175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:57.436823 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:57.436462    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:57.436823 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:57.436486    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:57.553628 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.553595    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" event={"ID":"ea8a1452-c2ec-449d-abf3-818d7fffb9d4","Type":"ContainerStarted","Data":"8526588d383d9e001b76b4b0e7fb02a9d540ed79b6a5d44f9c56fc40cc0ede40"}
Sep 08 15:17:57.553910 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.553887    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:57.555580 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.555555    2575 generic.go:356] "Generic (PLEG): container finished" podID="abd18e81-3dc2-4b04-b48e-f0ba03f796d1" containerID="3e66dcb92350926abc4184cf3bf29ddb2f70ba12d49b519ddf4077e02d4013b7" exitCode=0
Sep 08 15:17:57.555675 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.555602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerDied","Data":"3e66dcb92350926abc4184cf3bf29ddb2f70ba12d49b519ddf4077e02d4013b7"}
Sep 08 15:17:57.569808 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.569783    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:57.595062 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:57.595020    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" podStartSLOduration=8.305785338 podStartE2EDuration="26.595005943s" totalImagesPullingTime="18.289220605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:57.592302623 +0000 UTC m=+26.667594906" watchObservedRunningTime="2026-09-08 15:17:57.595005943 +0000 UTC m=+26.670298235"
Sep 08 15:17:58.435668 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.435642    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:58.435799 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:58.435776    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:58.560650 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.560573    2575 generic.go:356] "Generic (PLEG): container finished" podID="abd18e81-3dc2-4b04-b48e-f0ba03f796d1" containerID="6510e325089f9fad323bcb2460949f7178758bb20fb91c161c5d9c9865e3bb40" exitCode=0
Sep 08 15:17:58.561321 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.560656    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerDied","Data":"6510e325089f9fad323bcb2460949f7178758bb20fb91c161c5d9c9865e3bb40"}
Sep 08 15:17:58.563148 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.563118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg" event={"ID":"ed22bc20-2c4d-4c50-a0b3-ace51bdd9328","Type":"ContainerStarted","Data":"530e393d22c785d2f111a576067835aafb174d7e5b24b8a925d6f7a285519840"}
Sep 08 15:17:58.563280 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.563173    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:17:58.563644 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.563609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:58.578407 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.578389    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:17:58.645795 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.645756    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbpgg" podStartSLOduration=4.413830245 podStartE2EDuration="27.645741196s" totalImagesPullingTime="23.231910951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:58.645335045 +0000 UTC m=+27.720627355" watchObservedRunningTime="2026-09-08 15:17:58.645741196 +0000 UTC m=+27.721033499"
Sep 08 15:17:58.963588 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.963438    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m66rl"]
Sep 08 15:17:58.963708 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.963690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:17:58.963784 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:58.963767    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:17:58.966724 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.966702    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kmlj7"]
Sep 08 15:17:58.966862 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.966832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:17:58.967008 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:58.966982    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:17:58.967293 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.967273    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mbkb4"]
Sep 08 15:17:58.967391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:58.967377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:17:58.967496 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:17:58.967477    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:17:59.567570 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:59.567481    2575 generic.go:356] "Generic (PLEG): container finished" podID="abd18e81-3dc2-4b04-b48e-f0ba03f796d1" containerID="a72c9f0621d888436033a6b18674cfaa6c08c67699169014b2ad0b7edc14a881" exitCode=0
Sep 08 15:17:59.568155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:59.567563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerDied","Data":"a72c9f0621d888436033a6b18674cfaa6c08c67699169014b2ad0b7edc14a881"}
Sep 08 15:17:59.568155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:17:59.567615    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:18:00.435900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:00.435866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:00.435900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:00.435896    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:00.436095 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:00.435904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:00.436095 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:00.436025    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:18:00.436189 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:00.436144    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:18:00.436235 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:00.436217    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:18:00.569725 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:00.569699    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:18:02.435904 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.435868    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:02.435904 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.435896    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:02.436378 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:02.435983    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:18:02.436378 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.436027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:02.436378 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:02.436132    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:18:02.436378 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:02.436221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:18:02.921226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.921192    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:18:02.921453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.921438    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:18:02.937438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.937394    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" podUID="ea8a1452-c2ec-449d-abf3-818d7fffb9d4" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 08 15:18:02.947309 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:02.947277    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-l8264" podUID="ea8a1452-c2ec-449d-abf3-818d7fffb9d4" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 08 15:18:04.436213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:04.436183    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:04.436578 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:04.436219    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:04.436578 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:04.436297    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m66rl" podUID="37ac627f-21b3-469b-b8ed-478d372b7f0a"
Sep 08 15:18:04.436578 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:04.436339    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:04.436578 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:04.436427    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmlj7" podUID="4317d79b-6d07-4211-8ecc-3d56b8d5123a"
Sep 08 15:18:04.436578 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:04.436450    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbkb4" podUID="6e192123-3289-42af-92d3-98ecb3d76e43"
Sep 08 15:18:04.582104 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:04.582081    2575 generic.go:356] "Generic (PLEG): container finished" podID="abd18e81-3dc2-4b04-b48e-f0ba03f796d1" containerID="55d99933efb11ed213b1ca3c5139cab5a81563ccf301c2cb0f3fc0286598c537" exitCode=0
Sep 08 15:18:04.582217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:04.582123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerDied","Data":"55d99933efb11ed213b1ca3c5139cab5a81563ccf301c2cb0f3fc0286598c537"}
Sep 08 15:18:05.055136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.055069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:05.055236 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.055207    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:18:05.055276 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.055262    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret podName:37ac627f-21b3-469b-b8ed-478d372b7f0a nodeName:}" failed. No retries permitted until 2026-09-08 15:18:37.055246081 +0000 UTC m=+66.130538352 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret") pod "global-pull-secret-syncer-m66rl" (UID: "37ac627f-21b3-469b-b8ed-478d372b7f0a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:18:05.155726 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.155554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:05.155832 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.155681    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:18:05.155832 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.155771    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs podName:4317d79b-6d07-4211-8ecc-3d56b8d5123a nodeName:}" failed. No retries permitted until 2026-09-08 15:18:37.155756526 +0000 UTC m=+66.231048797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs") pod "network-metrics-daemon-kmlj7" (UID: "4317d79b-6d07-4211-8ecc-3d56b8d5123a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:18:05.256895 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.256868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:05.257007 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.256993    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:18:05.257048 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.257011    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:18:05.257048 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.257020    2575 projected.go:196] Error preparing data for projected volume kube-api-access-h74nz for pod openshift-network-diagnostics/network-check-target-mbkb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:18:05.257110 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:05.257063    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz podName:6e192123-3289-42af-92d3-98ecb3d76e43 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:37.257049261 +0000 UTC m=+66.332341549 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h74nz" (UniqueName: "kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz") pod "network-check-target-mbkb4" (UID: "6e192123-3289-42af-92d3-98ecb3d76e43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:18:05.587581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.587554    2575 generic.go:356] "Generic (PLEG): container finished" podID="abd18e81-3dc2-4b04-b48e-f0ba03f796d1" containerID="92c46d7591e5a5dfad923c7e16fcf6828e8639fa844e646e5c744d28edc10b97" exitCode=0
Sep 08 15:18:05.587961 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.587591    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerDied","Data":"92c46d7591e5a5dfad923c7e16fcf6828e8639fa844e646e5c744d28edc10b97"}
Sep 08 15:18:05.735453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.735408    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeReady"
Sep 08 15:18:05.735560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.735496    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 15:18:05.775028 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.775000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69fcd5c7-7p5vq"]
Sep 08 15:18:05.786084 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.786065    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.788885 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.788866    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-smd6z\""
Sep 08 15:18:05.788982 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.788897    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 15:18:05.789546 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.789525    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 15:18:05.790355 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.790338    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 15:18:05.790575 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.790561    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v9cbd"]
Sep 08 15:18:05.797000 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.796979    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 15:18:05.802300 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.802283    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lt4mz"]
Sep 08 15:18:05.802427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.802411    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:05.806724 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.806706    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 15:18:05.806823 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.806721    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65z9v\""
Sep 08 15:18:05.807867 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.806982    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 15:18:05.816004 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.815985    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69fcd5c7-7p5vq"]
Sep 08 15:18:05.816082 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.816015    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lt4mz"]
Sep 08 15:18:05.816124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.816083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:05.818312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.818293    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v9cbd"]
Sep 08 15:18:05.822365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.822054    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 15:18:05.822365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.822062    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 15:18:05.822365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.822313    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9zz4p\""
Sep 08 15:18:05.822700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.822678    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 15:18:05.962433 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgl6f\" (UniqueName: \"kubernetes.io/projected/a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6-kube-api-access-wgl6f\") pod \"ingress-canary-lt4mz\" (UID: \"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6\") " pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:05.962554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxrd\" (UniqueName: \"kubernetes.io/projected/379eddb9-5555-40ce-91d2-e88f787f7d27-kube-api-access-xxxrd\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:05.962554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-installation-pull-secrets\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cmw\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-kube-api-access-94cmw\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6-cert\") pod \"ingress-canary-lt4mz\" (UID: \"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6\") " pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:05.962554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379eddb9-5555-40ce-91d2-e88f787f7d27-tmp-dir\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:05.962765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-image-registry-private-configuration\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-certificates\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/379eddb9-5555-40ce-91d2-e88f787f7d27-config-volume\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:05.962765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:05.962765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b1467eb-adda-4980-b92f-c849f6fcf3a6-ca-trust-extracted\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962763    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-bound-sa-token\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:05.962987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:05.962797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-trusted-ca\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063561 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgl6f\" (UniqueName: \"kubernetes.io/projected/a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6-kube-api-access-wgl6f\") pod \"ingress-canary-lt4mz\" (UID: \"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6\") " pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:06.063561 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxrd\" (UniqueName: \"kubernetes.io/projected/379eddb9-5555-40ce-91d2-e88f787f7d27-kube-api-access-xxxrd\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-installation-pull-secrets\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cmw\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-kube-api-access-94cmw\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6-cert\") pod \"ingress-canary-lt4mz\" (UID: \"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6\") " pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379eddb9-5555-40ce-91d2-e88f787f7d27-tmp-dir\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-image-registry-private-configuration\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-certificates\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063727    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/379eddb9-5555-40ce-91d2-e88f787f7d27-config-volume\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b1467eb-adda-4980-b92f-c849f6fcf3a6-ca-trust-extracted\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-bound-sa-token\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.063836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.063868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-trusted-ca\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.063937    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.063962    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fcd5c7-7p5vq: secret "image-registry-tls" not found
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.064036    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls podName:3b1467eb-adda-4980-b92f-c849f6fcf3a6 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:06.564015094 +0000 UTC m=+35.639307382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls") pod "image-registry-69fcd5c7-7p5vq" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6") : secret "image-registry-tls" not found
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.064116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379eddb9-5555-40ce-91d2-e88f787f7d27-tmp-dir\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.064274    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b1467eb-adda-4980-b92f-c849f6fcf3a6-ca-trust-extracted\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.064294    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.064380    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls podName:379eddb9-5555-40ce-91d2-e88f787f7d27 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:06.564362351 +0000 UTC m=+35.639654635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls") pod "dns-default-v9cbd" (UID: "379eddb9-5555-40ce-91d2-e88f787f7d27") : secret "dns-default-metrics-tls" not found
Sep 08 15:18:06.064450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.064398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-certificates\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.064780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.064744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/379eddb9-5555-40ce-91d2-e88f787f7d27-config-volume\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.064998 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.064978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-trusted-ca\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.068079 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.068058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-image-registry-private-configuration\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.068174 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.068062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-installation-pull-secrets\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.068174 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.068135    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6-cert\") pod \"ingress-canary-lt4mz\" (UID: \"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6\") " pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:06.075012 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.074993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-bound-sa-token\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.075139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.075108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxrd\" (UniqueName: \"kubernetes.io/projected/379eddb9-5555-40ce-91d2-e88f787f7d27-kube-api-access-xxxrd\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.075383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.075363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgl6f\" (UniqueName: \"kubernetes.io/projected/a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6-kube-api-access-wgl6f\") pod \"ingress-canary-lt4mz\" (UID: \"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6\") " pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:06.075756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.075740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cmw\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-kube-api-access-94cmw\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.125132 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.125115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lt4mz"
Sep 08 15:18:06.429803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.429767    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lt4mz"]
Sep 08 15:18:06.432930 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:06.432906    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d7fb1d_6695_45af_ba39_0cb7ed49e6f6.slice/crio-ea18b2851e1ce4c1fed182b05df8ea19c3c4b7dfac5668e3bac35aaa3c3e530d WatchSource:0}: Error finding container ea18b2851e1ce4c1fed182b05df8ea19c3c4b7dfac5668e3bac35aaa3c3e530d: Status 404 returned error can't find the container with id ea18b2851e1ce4c1fed182b05df8ea19c3c4b7dfac5668e3bac35aaa3c3e530d
Sep 08 15:18:06.435305 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.435281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:06.435408 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.435390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:06.435535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.435507    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:06.440123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.439968    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 15:18:06.440123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.439969    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:18:06.440123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.440011    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:18:06.440123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.440047    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-76z9f\""
Sep 08 15:18:06.440123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.440047    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-np4gb\""
Sep 08 15:18:06.440123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.440076    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 15:18:06.567077 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.567057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:06.567216 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.567123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:06.567216 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.567210    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:18:06.567324 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.567275    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls podName:379eddb9-5555-40ce-91d2-e88f787f7d27 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:07.567259425 +0000 UTC m=+36.642551696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls") pod "dns-default-v9cbd" (UID: "379eddb9-5555-40ce-91d2-e88f787f7d27") : secret "dns-default-metrics-tls" not found
Sep 08 15:18:06.567324 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.567284    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:18:06.567324 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.567299    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fcd5c7-7p5vq: secret "image-registry-tls" not found
Sep 08 15:18:06.567493 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:06.567344    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls podName:3b1467eb-adda-4980-b92f-c849f6fcf3a6 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:07.56733179 +0000 UTC m=+36.642624061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls") pod "image-registry-69fcd5c7-7p5vq" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6") : secret "image-registry-tls" not found
Sep 08 15:18:06.593446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.593426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgclk" event={"ID":"abd18e81-3dc2-4b04-b48e-f0ba03f796d1","Type":"ContainerStarted","Data":"27514931a9f441c41f6ea454de624bd46d3c7e3cc1bd1e7abb17f78bdca25e0f"}
Sep 08 15:18:06.594683 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.594662    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lt4mz" event={"ID":"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6","Type":"ContainerStarted","Data":"ea18b2851e1ce4c1fed182b05df8ea19c3c4b7dfac5668e3bac35aaa3c3e530d"}
Sep 08 15:18:06.623467 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:06.621475    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jgclk" podStartSLOduration=7.224409967 podStartE2EDuration="35.621457493s" totalImagesPullingTime="28.397047526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:06.6204851 +0000 UTC m=+35.695777420" watchObservedRunningTime="2026-09-08 15:18:06.621457493 +0000 UTC m=+35.696749793"
Sep 08 15:18:07.574305 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:07.574272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:07.574522 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:07.574330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:07.574522 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:07.574409    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:18:07.574522 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:07.574423    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fcd5c7-7p5vq: secret "image-registry-tls" not found
Sep 08 15:18:07.574522 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:07.574427    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:18:07.574522 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:07.574474    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls podName:379eddb9-5555-40ce-91d2-e88f787f7d27 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:09.574460344 +0000 UTC m=+38.649752615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls") pod "dns-default-v9cbd" (UID: "379eddb9-5555-40ce-91d2-e88f787f7d27") : secret "dns-default-metrics-tls" not found
Sep 08 15:18:07.574522 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:07.574486    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls podName:3b1467eb-adda-4980-b92f-c849f6fcf3a6 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:09.574480183 +0000 UTC m=+38.649772454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls") pod "image-registry-69fcd5c7-7p5vq" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6") : secret "image-registry-tls" not found
Sep 08 15:18:09.585952 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:09.585713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:09.586410 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:09.585875    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:18:09.586410 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:09.586027    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fcd5c7-7p5vq: secret "image-registry-tls" not found
Sep 08 15:18:09.586410 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:09.586081    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls podName:3b1467eb-adda-4980-b92f-c849f6fcf3a6 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:13.586063177 +0000 UTC m=+42.661355448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls") pod "image-registry-69fcd5c7-7p5vq" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6") : secret "image-registry-tls" not found
Sep 08 15:18:09.586410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:09.586006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:09.586410 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:09.586107    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:18:09.586410 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:09.586178    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls podName:379eddb9-5555-40ce-91d2-e88f787f7d27 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:13.586159924 +0000 UTC m=+42.661452194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls") pod "dns-default-v9cbd" (UID: "379eddb9-5555-40ce-91d2-e88f787f7d27") : secret "dns-default-metrics-tls" not found
Sep 08 15:18:10.293807 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.293780    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-sr7vw"]
Sep 08 15:18:10.314417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.314388    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-sr7vw"]
Sep 08 15:18:10.314552 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.314484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:10.318281 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.318261    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hzjhx\""
Sep 08 15:18:10.318382 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.318314    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 15:18:10.319790 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.319770    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 15:18:10.491022 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.490997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:10.491164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.491091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48cc9678-ebe1-450d-8fe8-6c5f955298fb-nginx-conf\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:10.592352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.592290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:10.592710 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.592365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48cc9678-ebe1-450d-8fe8-6c5f955298fb-nginx-conf\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:10.592710 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:10.592472    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:18:10.592710 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:10.592562    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert podName:48cc9678-ebe1-450d-8fe8-6c5f955298fb nodeName:}" failed. No retries permitted until 2026-09-08 15:18:11.092539996 +0000 UTC m=+40.167832270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-sr7vw" (UID: "48cc9678-ebe1-450d-8fe8-6c5f955298fb") : secret "networking-console-plugin-cert" not found
Sep 08 15:18:10.593055 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:10.593035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48cc9678-ebe1-450d-8fe8-6c5f955298fb-nginx-conf\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:11.096506 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:11.096466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:11.096667 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:11.096602    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:18:11.096740 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:11.096682    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert podName:48cc9678-ebe1-450d-8fe8-6c5f955298fb nodeName:}" failed. No retries permitted until 2026-09-08 15:18:12.096651238 +0000 UTC m=+41.171943512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-sr7vw" (UID: "48cc9678-ebe1-450d-8fe8-6c5f955298fb") : secret "networking-console-plugin-cert" not found
Sep 08 15:18:12.103574 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.103537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:12.103940 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:12.103702    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:18:12.103940 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:12.103769    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert podName:48cc9678-ebe1-450d-8fe8-6c5f955298fb nodeName:}" failed. No retries permitted until 2026-09-08 15:18:14.10375362 +0000 UTC m=+43.179045914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-sr7vw" (UID: "48cc9678-ebe1-450d-8fe8-6c5f955298fb") : secret "networking-console-plugin-cert" not found
Sep 08 15:18:12.946894 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.946861    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-svxgm"]
Sep 08 15:18:12.965542 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.965520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svxgm"]
Sep 08 15:18:12.965667 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.965628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:12.972689 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.972662    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 15:18:12.972804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.972661    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 15:18:12.972804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.972724    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wln7f\""
Sep 08 15:18:12.972804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.972752    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 15:18:12.972804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:12.972780    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 15:18:13.111729 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.111697    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2862\" (UniqueName: \"kubernetes.io/projected/5058d606-9e4a-4d96-8293-23b0245cba81-kube-api-access-v2862\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.112063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.111740    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5058d606-9e4a-4d96-8293-23b0245cba81-crio-socket\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.112063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.111800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.112063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.111871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.112063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.111893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5058d606-9e4a-4d96-8293-23b0245cba81-data-volume\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213057 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.212999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2862\" (UniqueName: \"kubernetes.io/projected/5058d606-9e4a-4d96-8293-23b0245cba81-kube-api-access-v2862\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213057 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.213042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5058d606-9e4a-4d96-8293-23b0245cba81-crio-socket\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213183 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.213069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213183 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.213108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213183 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.213125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5058d606-9e4a-4d96-8293-23b0245cba81-data-volume\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213303 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.213215    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:18:13.213303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.213241    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5058d606-9e4a-4d96-8293-23b0245cba81-crio-socket\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.213303 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.213274    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:13.213417 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.213303    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:13.713282493 +0000 UTC m=+42.788574765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : configmap "kube-rbac-proxy" not found
Sep 08 15:18:13.213417 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.213331    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:13.713316804 +0000 UTC m=+42.788609140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:13.213417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.213393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5058d606-9e4a-4d96-8293-23b0245cba81-data-volume\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.234736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.234712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2862\" (UniqueName: \"kubernetes.io/projected/5058d606-9e4a-4d96-8293-23b0245cba81-kube-api-access-v2862\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.611610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.611544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lt4mz" event={"ID":"a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6","Type":"ContainerStarted","Data":"cd3a566874913d87f261a65ab53e5a7c7fd02a2796c43f3deead79b3e2338642"}
Sep 08 15:18:13.616086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.616063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:13.616159 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.616123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:13.616228 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.616209    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:18:13.616263 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.616235    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:18:13.616263 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.616246    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fcd5c7-7p5vq: secret "image-registry-tls" not found
Sep 08 15:18:13.616328 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.616286    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls podName:379eddb9-5555-40ce-91d2-e88f787f7d27 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:21.616268238 +0000 UTC m=+50.691560509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls") pod "dns-default-v9cbd" (UID: "379eddb9-5555-40ce-91d2-e88f787f7d27") : secret "dns-default-metrics-tls" not found
Sep 08 15:18:13.616328 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.616303    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls podName:3b1467eb-adda-4980-b92f-c849f6fcf3a6 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:21.61629636 +0000 UTC m=+50.691588631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls") pod "image-registry-69fcd5c7-7p5vq" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6") : secret "image-registry-tls" not found
Sep 08 15:18:13.629516 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.629458    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lt4mz" podStartSLOduration=2.554816572 podStartE2EDuration="8.629441471s" totalImagesPullingTime="6.074624899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:13.629440395 +0000 UTC m=+42.704732704" watchObservedRunningTime="2026-09-08 15:18:13.629441471 +0000 UTC m=+42.704733767"
Sep 08 15:18:13.716678 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.716643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.716781 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:13.716722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:13.716879 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.716801    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:18:13.716879 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.716879    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:14.716862593 +0000 UTC m=+43.792154865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : configmap "kube-rbac-proxy" not found
Sep 08 15:18:13.716991 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.716883    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:13.716991 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:13.716938    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:14.716918235 +0000 UTC m=+43.792210524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:14.119071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.119040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:14.119396 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:14.119178    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:18:14.119396 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:14.119230    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert podName:48cc9678-ebe1-450d-8fe8-6c5f955298fb nodeName:}" failed. No retries permitted until 2026-09-08 15:18:18.11921729 +0000 UTC m=+47.194509560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-sr7vw" (UID: "48cc9678-ebe1-450d-8fe8-6c5f955298fb") : secret "networking-console-plugin-cert" not found
Sep 08 15:18:14.268282 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.268259    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"]
Sep 08 15:18:14.288366 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.288346    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"]
Sep 08 15:18:14.288459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.288435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"
Sep 08 15:18:14.291505 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.291487    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 15:18:14.291614 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.291597    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ph5cc\""
Sep 08 15:18:14.292707 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.292688    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 15:18:14.390457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.390438    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bc6nc_c094bb8e-9ba5-40ad-97c5-3d236e59f365/dns-node-resolver/0.log"
Sep 08 15:18:14.421730 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.421705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q89p5\" (UniqueName: \"kubernetes.io/projected/342f7926-a759-4ae1-a2d8-a58500895217-kube-api-access-q89p5\") pod \"migrator-fdbb8fdff-4t8ck\" (UID: \"342f7926-a759-4ae1-a2d8-a58500895217\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"
Sep 08 15:18:14.522497 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.522469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q89p5\" (UniqueName: \"kubernetes.io/projected/342f7926-a759-4ae1-a2d8-a58500895217-kube-api-access-q89p5\") pod \"migrator-fdbb8fdff-4t8ck\" (UID: \"342f7926-a759-4ae1-a2d8-a58500895217\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"
Sep 08 15:18:14.531293 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.531270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q89p5\" (UniqueName: \"kubernetes.io/projected/342f7926-a759-4ae1-a2d8-a58500895217-kube-api-access-q89p5\") pod \"migrator-fdbb8fdff-4t8ck\" (UID: \"342f7926-a759-4ae1-a2d8-a58500895217\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"
Sep 08 15:18:14.597560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.597536    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"
Sep 08 15:18:14.724349 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.724320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:14.724490 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.724385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:14.724490 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:14.724454    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:18:14.724601 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:14.724518    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:16.724502597 +0000 UTC m=+45.799794887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : configmap "kube-rbac-proxy" not found
Sep 08 15:18:14.724601 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:14.724542    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:14.724601 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:14.724600    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:16.724585793 +0000 UTC m=+45.799878075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:14.883405 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:14.883385    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck"]
Sep 08 15:18:14.885572 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:14.885554    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f7926_a759_4ae1_a2d8_a58500895217.slice/crio-c625267c8225bd01aba8862ae7f41e58aefa3a7ad031de81874fa26500345981 WatchSource:0}: Error finding container c625267c8225bd01aba8862ae7f41e58aefa3a7ad031de81874fa26500345981: Status 404 returned error can't find the container with id c625267c8225bd01aba8862ae7f41e58aefa3a7ad031de81874fa26500345981
Sep 08 15:18:15.555093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.555067    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6mnzd"]
Sep 08 15:18:15.573033 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.573009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6mnzd"]
Sep 08 15:18:15.573153 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.573095    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.577226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.577197    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 15:18:15.577344 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.577205    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 15:18:15.578286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.578268    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fmg9p\""
Sep 08 15:18:15.578370 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.578269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 15:18:15.578370 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.578269    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 15:18:15.584811 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.584793    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gzb7q_eff82d57-cf61-4995-8dd9-d3da3aa6269d/node-ca/0.log"
Sep 08 15:18:15.617009 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.616982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck" event={"ID":"342f7926-a759-4ae1-a2d8-a58500895217","Type":"ContainerStarted","Data":"c625267c8225bd01aba8862ae7f41e58aefa3a7ad031de81874fa26500345981"}
Sep 08 15:18:15.732321 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.732291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-signing-key\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.732426 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.732352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhh9\" (UniqueName: \"kubernetes.io/projected/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-kube-api-access-xnhh9\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.732426 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.732415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-signing-cabundle\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.833248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.833179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-signing-key\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.833248 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.833231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhh9\" (UniqueName: \"kubernetes.io/projected/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-kube-api-access-xnhh9\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.833427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.833295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-signing-cabundle\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.834076 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.834053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-signing-cabundle\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.836318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.836297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-signing-key\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.845073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.845049    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhh9\" (UniqueName: \"kubernetes.io/projected/5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252-kube-api-access-xnhh9\") pod \"service-ca-75d99c484-6mnzd\" (UID: \"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252\") " pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:15.885173 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:15.885152    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-6mnzd"
Sep 08 15:18:16.188044 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:16.188012    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-6mnzd"]
Sep 08 15:18:16.190510 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:16.190492    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lt4mz_a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6/serve-healthcheck-canary/0.log"
Sep 08 15:18:16.337366 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:16.337332    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddf5a7e_1521_4fd6_9f1c_e7ed0d6e2252.slice/crio-70f494fdad4e8700a71f3979db20e44bd39e8aafa3a6a24e0e8fe462a55c9117 WatchSource:0}: Error finding container 70f494fdad4e8700a71f3979db20e44bd39e8aafa3a6a24e0e8fe462a55c9117: Status 404 returned error can't find the container with id 70f494fdad4e8700a71f3979db20e44bd39e8aafa3a6a24e0e8fe462a55c9117
Sep 08 15:18:16.620621 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:16.620564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-6mnzd" event={"ID":"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252","Type":"ContainerStarted","Data":"70f494fdad4e8700a71f3979db20e44bd39e8aafa3a6a24e0e8fe462a55c9117"}
Sep 08 15:18:16.740198 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:16.740174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:16.740280 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:16.740250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:16.740322 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:16.740309    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:16.740322 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:16.740319    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:18:16.740397 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:16.740366    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:20.740349288 +0000 UTC m=+49.815641559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:16.740397 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:16.740379    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:20.740372784 +0000 UTC m=+49.815665055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : configmap "kube-rbac-proxy" not found
Sep 08 15:18:17.625566 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:17.625526    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck" event={"ID":"342f7926-a759-4ae1-a2d8-a58500895217","Type":"ContainerStarted","Data":"567abf507e2cf8d3eb97a724da5cdecb84062daf95df642576ca2cb2e63f03b3"}
Sep 08 15:18:17.626009 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:17.625586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck" event={"ID":"342f7926-a759-4ae1-a2d8-a58500895217","Type":"ContainerStarted","Data":"77fcbd2245b671c769927148e12f6569868c6f06360006065de8364eaa27dcad"}
Sep 08 15:18:17.644632 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:17.644590    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4t8ck" podStartSLOduration=1.844197968 podStartE2EDuration="3.644575078s" totalImagesPullingTime="1.80037711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:17.644214217 +0000 UTC m=+46.719506528" watchObservedRunningTime="2026-09-08 15:18:17.644575078 +0000 UTC m=+46.719867371"
Sep 08 15:18:18.152831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:18.152799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:18.153018 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:18.152952    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:18:18.153068 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:18.153019    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert podName:48cc9678-ebe1-450d-8fe8-6c5f955298fb nodeName:}" failed. No retries permitted until 2026-09-08 15:18:26.153002741 +0000 UTC m=+55.228295013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-sr7vw" (UID: "48cc9678-ebe1-450d-8fe8-6c5f955298fb") : secret "networking-console-plugin-cert" not found
Sep 08 15:18:19.632254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:19.632185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-6mnzd" event={"ID":"5ddf5a7e-1521-4fd6-9f1c-e7ed0d6e2252","Type":"ContainerStarted","Data":"3239f3216a724438bd57cf77a036e56861d2a5b5751f1b412aae48dc44182d77"}
Sep 08 15:18:19.651336 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:19.651293    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-6mnzd" podStartSLOduration=1.8255086870000001 podStartE2EDuration="4.651277442s" totalImagesPullingTime="2.825768755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:19.650101735 +0000 UTC m=+48.725394047" watchObservedRunningTime="2026-09-08 15:18:19.651277442 +0000 UTC m=+48.726569734"
Sep 08 15:18:20.770749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:20.770716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:20.771243 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:20.770881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:20.771719 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:20.771696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5058d606-9e4a-4d96-8293-23b0245cba81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:20.771803 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:20.771789    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:20.771887 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:20.771876    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls podName:5058d606-9e4a-4d96-8293-23b0245cba81 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:28.771836755 +0000 UTC m=+57.847129033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svxgm" (UID: "5058d606-9e4a-4d96-8293-23b0245cba81") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:18:21.676702 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:21.676660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:21.676903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:21.676749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:21.676903 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:21.676800    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:18:21.676903 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:21.676815    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fcd5c7-7p5vq: secret "image-registry-tls" not found
Sep 08 15:18:21.676903 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:21.676858    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:18:21.676903 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:21.676871    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls podName:3b1467eb-adda-4980-b92f-c849f6fcf3a6 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:37.676857729 +0000 UTC m=+66.752150020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls") pod "image-registry-69fcd5c7-7p5vq" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6") : secret "image-registry-tls" not found
Sep 08 15:18:21.677118 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:21.676916    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls podName:379eddb9-5555-40ce-91d2-e88f787f7d27 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:37.676898106 +0000 UTC m=+66.752190376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls") pod "dns-default-v9cbd" (UID: "379eddb9-5555-40ce-91d2-e88f787f7d27") : secret "dns-default-metrics-tls" not found
Sep 08 15:18:26.215560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:26.215519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:26.216084 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:26.215634    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:18:26.216084 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:26.215695    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert podName:48cc9678-ebe1-450d-8fe8-6c5f955298fb nodeName:}" failed. No retries permitted until 2026-09-08 15:18:42.215678309 +0000 UTC m=+71.290970644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-sr7vw" (UID: "48cc9678-ebe1-450d-8fe8-6c5f955298fb") : secret "networking-console-plugin-cert" not found
Sep 08 15:18:28.835175 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:28.835138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:28.837346 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:28.837328    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5058d606-9e4a-4d96-8293-23b0245cba81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svxgm\" (UID: \"5058d606-9e4a-4d96-8293-23b0245cba81\") " pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:28.874320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:28.874296    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svxgm"
Sep 08 15:18:29.147952 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:29.147922    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svxgm"]
Sep 08 15:18:29.661722 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:29.661689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svxgm" event={"ID":"5058d606-9e4a-4d96-8293-23b0245cba81","Type":"ContainerStarted","Data":"36e8c7461f96e87603df58254a373b4e7720cd22252932e49f67e9e92eee68be"}
Sep 08 15:18:29.661890 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:29.661776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svxgm" event={"ID":"5058d606-9e4a-4d96-8293-23b0245cba81","Type":"ContainerStarted","Data":"ee2b07866db6d533fe711561832155ff858f5d778832d096c2aeef6380119c12"}
Sep 08 15:18:30.669789 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:30.669759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svxgm" event={"ID":"5058d606-9e4a-4d96-8293-23b0245cba81","Type":"ContainerStarted","Data":"27dd942b8b816569fad185bcc1ee357a0713ffc7c3d67f9a8c570b91e3acf84e"}
Sep 08 15:18:31.676069 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:31.675996    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svxgm" event={"ID":"5058d606-9e4a-4d96-8293-23b0245cba81","Type":"ContainerStarted","Data":"b44b94a269761da6bee4c9d3ee629a6f2a014399ceb919b8fe3a9a548d9257f4"}
Sep 08 15:18:31.700258 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:31.700208    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-svxgm" podStartSLOduration=17.551285996 podStartE2EDuration="19.700192447s" totalImagesPullingTime="2.148906451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:18:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:31.698160003 +0000 UTC m=+60.773452316" watchObservedRunningTime="2026-09-08 15:18:31.700192447 +0000 UTC m=+60.775484740"
Sep 08 15:18:32.948456 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:32.948428    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8264"
Sep 08 15:18:35.936192 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.936147    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"]
Sep 08 15:18:35.938771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.938750    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"]
Sep 08 15:18:35.938920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.938898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:35.940812 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.940787    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:35.942140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.942115    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 15:18:35.942265 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.942173    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 15:18:35.943293 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.943269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 15:18:35.943410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.943269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 15:18:35.944506 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.944484    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 15:18:35.946383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.946092    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bxh5k\""
Sep 08 15:18:35.951233 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.951212    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"]
Sep 08 15:18:35.952051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.952028    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"]
Sep 08 15:18:35.972882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.972860    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69fcd5c7-7p5vq"]
Sep 08 15:18:35.973038 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:35.973017    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq" podUID="3b1467eb-adda-4980-b92f-c849f6fcf3a6"
Sep 08 15:18:35.986882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.986838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwrg\" (UniqueName: \"kubernetes.io/projected/c18ac297-9774-495a-a11e-ca82ef5e3407-kube-api-access-bjwrg\") pod \"managed-serviceaccount-addon-agent-55d86d8f64-s8wfw\" (UID: \"c18ac297-9774-495a-a11e-ca82ef5e3407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:35.986968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.986887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:35.986968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.986944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znfv\" (UniqueName: \"kubernetes.io/projected/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-kube-api-access-6znfv\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:35.987034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.987018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-tmp\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:35.987074 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:35.987048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c18ac297-9774-495a-a11e-ca82ef5e3407-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55d86d8f64-s8wfw\" (UID: \"c18ac297-9774-495a-a11e-ca82ef5e3407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:36.087907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.087825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-tmp\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.088028 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.087957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c18ac297-9774-495a-a11e-ca82ef5e3407-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55d86d8f64-s8wfw\" (UID: \"c18ac297-9774-495a-a11e-ca82ef5e3407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:36.088028 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.088013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwrg\" (UniqueName: \"kubernetes.io/projected/c18ac297-9774-495a-a11e-ca82ef5e3407-kube-api-access-bjwrg\") pod \"managed-serviceaccount-addon-agent-55d86d8f64-s8wfw\" (UID: \"c18ac297-9774-495a-a11e-ca82ef5e3407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:36.088137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.088049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.088137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.088100    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znfv\" (UniqueName: \"kubernetes.io/projected/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-kube-api-access-6znfv\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.088240 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.088149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-tmp\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.090631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.090604    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.090709 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.090682    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c18ac297-9774-495a-a11e-ca82ef5e3407-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55d86d8f64-s8wfw\" (UID: \"c18ac297-9774-495a-a11e-ca82ef5e3407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:36.109333 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.109309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwrg\" (UniqueName: \"kubernetes.io/projected/c18ac297-9774-495a-a11e-ca82ef5e3407-kube-api-access-bjwrg\") pod \"managed-serviceaccount-addon-agent-55d86d8f64-s8wfw\" (UID: \"c18ac297-9774-495a-a11e-ca82ef5e3407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:36.109536 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.109514    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znfv\" (UniqueName: \"kubernetes.io/projected/2dc49f36-75b4-498f-84a6-1f1a726eb7a8-kube-api-access-6znfv\") pod \"klusterlet-addon-workmgr-7f8456bf4d-wtww8\" (UID: \"2dc49f36-75b4-498f-84a6-1f1a726eb7a8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.250731 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.250674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:36.268592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.268573    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"
Sep 08 15:18:36.523670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.523608    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"]
Sep 08 15:18:36.527236 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:36.527209    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc49f36_75b4_498f_84a6_1f1a726eb7a8.slice/crio-54970c8a73607237bed1ae7e9b102559df4800c33eafa8b867f57e42202ab9ac WatchSource:0}: Error finding container 54970c8a73607237bed1ae7e9b102559df4800c33eafa8b867f57e42202ab9ac: Status 404 returned error can't find the container with id 54970c8a73607237bed1ae7e9b102559df4800c33eafa8b867f57e42202ab9ac
Sep 08 15:18:36.545715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.545675    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw"]
Sep 08 15:18:36.550204 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:36.550181    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc18ac297_9774_495a_a11e_ca82ef5e3407.slice/crio-2f66cc0d6da7f248d7aa3e378e0eab0c6540359c1e46e0f8684db228991f492a WatchSource:0}: Error finding container 2f66cc0d6da7f248d7aa3e378e0eab0c6540359c1e46e0f8684db228991f492a: Status 404 returned error can't find the container with id 2f66cc0d6da7f248d7aa3e378e0eab0c6540359c1e46e0f8684db228991f492a
Sep 08 15:18:36.692100 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.692072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw" event={"ID":"c18ac297-9774-495a-a11e-ca82ef5e3407","Type":"ContainerStarted","Data":"2f66cc0d6da7f248d7aa3e378e0eab0c6540359c1e46e0f8684db228991f492a"}
Sep 08 15:18:36.693349 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.693332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:36.693432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.693350    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8" event={"ID":"2dc49f36-75b4-498f-84a6-1f1a726eb7a8","Type":"ContainerStarted","Data":"54970c8a73607237bed1ae7e9b102559df4800c33eafa8b867f57e42202ab9ac"}
Sep 08 15:18:36.699321 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.699303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:36.793130 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-bound-sa-token\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793130 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793117    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-installation-pull-secrets\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793253 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793136    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b1467eb-adda-4980-b92f-c849f6fcf3a6-ca-trust-extracted\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793253 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793153    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-trusted-ca\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793253 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793182    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-image-registry-private-configuration\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793253 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793238    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94cmw\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-kube-api-access-94cmw\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793282    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-certificates\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:36.793467 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793446    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1467eb-adda-4980-b92f-c849f6fcf3a6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:36.793607 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793581    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:18:36.793736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.793715    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:18:36.795400 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.795381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:36.795474 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.795430    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-kube-api-access-94cmw" (OuterVolumeSpecName: "kube-api-access-94cmw") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "kube-api-access-94cmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:36.795798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.795772    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:36.795798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.795789    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:36.894244 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894213    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-image-registry-private-configuration\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:36.894244 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894240    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94cmw\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-kube-api-access-94cmw\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:36.894371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894251    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-certificates\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:36.894371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894262    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-bound-sa-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:36.894371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894271    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b1467eb-adda-4980-b92f-c849f6fcf3a6-installation-pull-secrets\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:36.894371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894290    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b1467eb-adda-4980-b92f-c849f6fcf3a6-ca-trust-extracted\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:36.894371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:36.894298    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b1467eb-adda-4980-b92f-c849f6fcf3a6-trusted-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:37.095424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.095365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:37.098664 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.098644    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:18:37.108736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.108707    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ac627f-21b3-469b-b8ed-478d372b7f0a-original-pull-secret\") pod \"global-pull-secret-syncer-m66rl\" (UID: \"37ac627f-21b3-469b-b8ed-478d372b7f0a\") " pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:37.196117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.196090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:37.198968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.198945    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:18:37.209364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.209341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4317d79b-6d07-4211-8ecc-3d56b8d5123a-metrics-certs\") pod \"network-metrics-daemon-kmlj7\" (UID: \"4317d79b-6d07-4211-8ecc-3d56b8d5123a\") " pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:37.297064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.297033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:37.300144 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.300121    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 15:18:37.310585 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.310564    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 15:18:37.321276 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.321251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74nz\" (UniqueName: \"kubernetes.io/projected/6e192123-3289-42af-92d3-98ecb3d76e43-kube-api-access-h74nz\") pod \"network-check-target-mbkb4\" (UID: \"6e192123-3289-42af-92d3-98ecb3d76e43\") " pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:37.347935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.347882    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-np4gb\""
Sep 08 15:18:37.351483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.351457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m66rl"
Sep 08 15:18:37.355258 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.355238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:37.359181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.359163    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-76z9f\""
Sep 08 15:18:37.368570 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.368547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmlj7"
Sep 08 15:18:37.642226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.642193    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m66rl"]
Sep 08 15:18:37.645409 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:37.645380    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ac627f_21b3_469b_b8ed_478d372b7f0a.slice/crio-4ac71687d7fe371698b5c68f7a36a7f6f85ac0f356a68584769d315c3d9177e5 WatchSource:0}: Error finding container 4ac71687d7fe371698b5c68f7a36a7f6f85ac0f356a68584769d315c3d9177e5: Status 404 returned error can't find the container with id 4ac71687d7fe371698b5c68f7a36a7f6f85ac0f356a68584769d315c3d9177e5
Sep 08 15:18:37.667921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.667898    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kmlj7"]
Sep 08 15:18:37.669209 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.668868    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mbkb4"]
Sep 08 15:18:37.670238 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:37.670213    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e192123_3289_42af_92d3_98ecb3d76e43.slice/crio-4b4fc2c7bce7a7396d6a4b1de1a3b2fa09fdbfbd2ec1ad71ecc02bc144cc175c WatchSource:0}: Error finding container 4b4fc2c7bce7a7396d6a4b1de1a3b2fa09fdbfbd2ec1ad71ecc02bc144cc175c: Status 404 returned error can't find the container with id 4b4fc2c7bce7a7396d6a4b1de1a3b2fa09fdbfbd2ec1ad71ecc02bc144cc175c
Sep 08 15:18:37.671069 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:37.671043    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4317d79b_6d07_4211_8ecc_3d56b8d5123a.slice/crio-2c362412314c8811501d95a32ada6fce026a4c22ed571c6ce6e6b460ebad26f6 WatchSource:0}: Error finding container 2c362412314c8811501d95a32ada6fce026a4c22ed571c6ce6e6b460ebad26f6: Status 404 returned error can't find the container with id 2c362412314c8811501d95a32ada6fce026a4c22ed571c6ce6e6b460ebad26f6
Sep 08 15:18:37.698740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.698711    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m66rl" event={"ID":"37ac627f-21b3-469b-b8ed-478d372b7f0a","Type":"ContainerStarted","Data":"4ac71687d7fe371698b5c68f7a36a7f6f85ac0f356a68584769d315c3d9177e5"}
Sep 08 15:18:37.699741 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.699311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:37.699741 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.699356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:37.700751 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.700668    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmlj7" event={"ID":"4317d79b-6d07-4211-8ecc-3d56b8d5123a","Type":"ContainerStarted","Data":"2c362412314c8811501d95a32ada6fce026a4c22ed571c6ce6e6b460ebad26f6"}
Sep 08 15:18:37.702273 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.702245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"image-registry-69fcd5c7-7p5vq\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") " pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:37.702460 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.702437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mbkb4" event={"ID":"6e192123-3289-42af-92d3-98ecb3d76e43","Type":"ContainerStarted","Data":"4b4fc2c7bce7a7396d6a4b1de1a3b2fa09fdbfbd2ec1ad71ecc02bc144cc175c"}
Sep 08 15:18:37.702524 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.702468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fcd5c7-7p5vq"
Sep 08 15:18:37.703018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.702997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/379eddb9-5555-40ce-91d2-e88f787f7d27-metrics-tls\") pod \"dns-default-v9cbd\" (UID: \"379eddb9-5555-40ce-91d2-e88f787f7d27\") " pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:37.800600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.800568    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") pod \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\" (UID: \"3b1467eb-adda-4980-b92f-c849f6fcf3a6\") "
Sep 08 15:18:37.803314 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.803267    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3b1467eb-adda-4980-b92f-c849f6fcf3a6" (UID: "3b1467eb-adda-4980-b92f-c849f6fcf3a6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:37.901437 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.901356    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b1467eb-adda-4980-b92f-c849f6fcf3a6-registry-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:18:37.914314 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.914285    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65z9v\""
Sep 08 15:18:37.921700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:37.921674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:38.042257 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:38.042225    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69fcd5c7-7p5vq"]
Sep 08 15:18:38.045891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:38.045862    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69fcd5c7-7p5vq"]
Sep 08 15:18:38.210915 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:38.210817    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v9cbd"]
Sep 08 15:18:38.216522 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:38.216492    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379eddb9_5555_40ce_91d2_e88f787f7d27.slice/crio-1bc15d21bc7c93a6c4a196ff7bb387c028dd4797397cb5af47eeea3f1ab14f2e WatchSource:0}: Error finding container 1bc15d21bc7c93a6c4a196ff7bb387c028dd4797397cb5af47eeea3f1ab14f2e: Status 404 returned error can't find the container with id 1bc15d21bc7c93a6c4a196ff7bb387c028dd4797397cb5af47eeea3f1ab14f2e
Sep 08 15:18:38.709610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:38.709528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v9cbd" event={"ID":"379eddb9-5555-40ce-91d2-e88f787f7d27","Type":"ContainerStarted","Data":"1bc15d21bc7c93a6c4a196ff7bb387c028dd4797397cb5af47eeea3f1ab14f2e"}
Sep 08 15:18:39.440644 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:39.440609    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1467eb-adda-4980-b92f-c849f6fcf3a6" path="/var/lib/kubelet/pods/3b1467eb-adda-4980-b92f-c849f6fcf3a6/volumes"
Sep 08 15:18:42.246227 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:42.246177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:42.249256 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:42.249222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48cc9678-ebe1-450d-8fe8-6c5f955298fb-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-sr7vw\" (UID: \"48cc9678-ebe1-450d-8fe8-6c5f955298fb\") " pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:42.426271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:42.426224    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hzjhx\""
Sep 08 15:18:42.433723 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:42.433695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw"
Sep 08 15:18:45.611008 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.610657    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-sr7vw"]
Sep 08 15:18:45.613504 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:45.613478    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cc9678_ebe1_450d_8fe8_6c5f955298fb.slice/crio-5435ee86a0fc77e6357535f0f922551de004851fee98e783cee021d5ef998e2d WatchSource:0}: Error finding container 5435ee86a0fc77e6357535f0f922551de004851fee98e783cee021d5ef998e2d: Status 404 returned error can't find the container with id 5435ee86a0fc77e6357535f0f922551de004851fee98e783cee021d5ef998e2d
Sep 08 15:18:45.739032 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.739001    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw" event={"ID":"48cc9678-ebe1-450d-8fe8-6c5f955298fb","Type":"ContainerStarted","Data":"5435ee86a0fc77e6357535f0f922551de004851fee98e783cee021d5ef998e2d"}
Sep 08 15:18:45.740982 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.740954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw" event={"ID":"c18ac297-9774-495a-a11e-ca82ef5e3407","Type":"ContainerStarted","Data":"87e75439c6c28c72b38749b07fa4cc69447ececc0d863962bf9d8471993504df"}
Sep 08 15:18:45.742866 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.742825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8" event={"ID":"2dc49f36-75b4-498f-84a6-1f1a726eb7a8","Type":"ContainerStarted","Data":"c801c65ff77164cbcab758d2ada40c508149718f2db24581dc956f73e398f239"}
Sep 08 15:18:45.743024 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.743007    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:45.745024 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.744897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmlj7" event={"ID":"4317d79b-6d07-4211-8ecc-3d56b8d5123a","Type":"ContainerStarted","Data":"cdb5d7d5c1ad7d8f9ebc0eb689e004dd8b64fef2cc1896eaa0d4cf226f75a722"}
Sep 08 15:18:45.745024 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.744941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmlj7" event={"ID":"4317d79b-6d07-4211-8ecc-3d56b8d5123a","Type":"ContainerStarted","Data":"04154a9b312428991616d2f8d2a976067033197e2c3dbc60b37b28b5858dc33d"}
Sep 08 15:18:45.745024 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.744998    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8"
Sep 08 15:18:45.746318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.746300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mbkb4" event={"ID":"6e192123-3289-42af-92d3-98ecb3d76e43","Type":"ContainerStarted","Data":"a42bec223556876754b129bfa1b585d7f9a57b9fe16b7a41d05cb0f3cd636354"}
Sep 08 15:18:45.746421 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.746382    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:18:45.747899 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.747875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v9cbd" event={"ID":"379eddb9-5555-40ce-91d2-e88f787f7d27","Type":"ContainerStarted","Data":"d14ae14ac1e8b302be90e2cb60dd57cbc0603331ff8facac88f18779f81b3d87"}
Sep 08 15:18:45.748021 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.747920    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v9cbd" event={"ID":"379eddb9-5555-40ce-91d2-e88f787f7d27","Type":"ContainerStarted","Data":"3be14d55970a8e1b50a7834f8cab99bec863efbb0a7bb30bf0da01857e633fad"}
Sep 08 15:18:45.748021 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.747941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:45.749287 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.749268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m66rl" event={"ID":"37ac627f-21b3-469b-b8ed-478d372b7f0a","Type":"ContainerStarted","Data":"89c01d7df1964b92234c5e48bc4491debd489de8497363d236ced35dd964fb65"}
Sep 08 15:18:45.761672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.761624    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55d86d8f64-s8wfw" podStartSLOduration=1.9866369160000001 podStartE2EDuration="10.761609583s" totalImagesPullingTime="8.774972667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:45.760696831 +0000 UTC m=+74.835989124" watchObservedRunningTime="2026-09-08 15:18:45.761609583 +0000 UTC m=+74.836901870"
Sep 08 15:18:45.780831 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.780795    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mbkb4" podStartSLOduration=67.067373409 podStartE2EDuration="1m14.780783521s" totalImagesPullingTime="7.713410112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:45.780637158 +0000 UTC m=+74.855929451" watchObservedRunningTime="2026-09-08 15:18:45.780783521 +0000 UTC m=+74.856075813"
Sep 08 15:18:45.798791 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.798748    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v9cbd" podStartSLOduration=33.690606979 podStartE2EDuration="40.798730397s" totalImagesPullingTime="7.108123418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:45.798502698 +0000 UTC m=+74.873795010" watchObservedRunningTime="2026-09-08 15:18:45.798730397 +0000 UTC m=+74.874022692"
Sep 08 15:18:45.831485 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.831446    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f8456bf4d-wtww8" podStartSLOduration=2.034223878 podStartE2EDuration="10.831433237s" totalImagesPullingTime="8.797209359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:45.81611475 +0000 UTC m=+74.891407061" watchObservedRunningTime="2026-09-08 15:18:45.831433237 +0000 UTC m=+74.906725545"
Sep 08 15:18:45.831732 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.831706    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m66rl" podStartSLOduration=67.118783649 podStartE2EDuration="1m14.831699035s" totalImagesPullingTime="7.712915386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:45.831017167 +0000 UTC m=+74.906309461" watchObservedRunningTime="2026-09-08 15:18:45.831699035 +0000 UTC m=+74.906991328"
Sep 08 15:18:45.857862 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:45.857778    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kmlj7" podStartSLOduration=67.214332134 podStartE2EDuration="1m14.857766243s" totalImagesPullingTime="7.643434109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:45.857623898 +0000 UTC m=+74.932916191" watchObservedRunningTime="2026-09-08 15:18:45.857766243 +0000 UTC m=+74.933058535"
Sep 08 15:18:47.590606 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.590562    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8fw8v"]
Sep 08 15:18:47.594165 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.594134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.598427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.597496    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 15:18:47.598427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.597749    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 15:18:47.598427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.598047    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 15:18:47.598427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.598276    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 15:18:47.599377 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.599354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 15:18:47.599989 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.599962    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fp5sn\""
Sep 08 15:18:47.600292 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.600277    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 15:18:47.688169 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zmb\" (UniqueName: \"kubernetes.io/projected/d77f885d-cd3e-4af3-ad90-adce419f8b25-kube-api-access-65zmb\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688270 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688270 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688270 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-tls\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-root\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-wtmp\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-sys\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-textfile\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.688608 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.688436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77f885d-cd3e-4af3-ad90-adce419f8b25-metrics-client-ca\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.758466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.758398    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw" event={"ID":"48cc9678-ebe1-450d-8fe8-6c5f955298fb","Type":"ContainerStarted","Data":"c5686627ec2f8ce08e679040adebc3a8846f3afbb4132eb2c63b6cd076a0dc96"}
Sep 08 15:18:47.778858 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.778798    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-sr7vw" podStartSLOduration=36.309973356 podStartE2EDuration="37.778782809s" totalImagesPullingTime="1.468809453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:47.778186218 +0000 UTC m=+76.853478512" watchObservedRunningTime="2026-09-08 15:18:47.778782809 +0000 UTC m=+76.854075105"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zmb\" (UniqueName: \"kubernetes.io/projected/d77f885d-cd3e-4af3-ad90-adce419f8b25-kube-api-access-65zmb\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-tls\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-root\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-wtmp\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-sys\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-textfile\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77f885d-cd3e-4af3-ad90-adce419f8b25-metrics-client-ca\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.789741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-root\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.790212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77f885d-cd3e-4af3-ad90-adce419f8b25-metrics-client-ca\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.790357    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.790431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-sys\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.790592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-textfile\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.790807 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:47.790684    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 15:18:47.790807 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:18:47.790738    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-tls podName:d77f885d-cd3e-4af3-ad90-adce419f8b25 nodeName:}" failed. No retries permitted until 2026-09-08 15:18:48.29072018 +0000 UTC m=+77.366012471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-tls") pod "node-exporter-8fw8v" (UID: "d77f885d-cd3e-4af3-ad90-adce419f8b25") : secret "node-exporter-tls" not found
Sep 08 15:18:47.790807 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.790359    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-wtmp\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.800894 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.800873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:47.808611 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:47.808582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zmb\" (UniqueName: \"kubernetes.io/projected/d77f885d-cd3e-4af3-ad90-adce419f8b25-kube-api-access-65zmb\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:48.293270 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:48.293243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-tls\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:48.295453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:48.295427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77f885d-cd3e-4af3-ad90-adce419f8b25-node-exporter-tls\") pod \"node-exporter-8fw8v\" (UID: \"d77f885d-cd3e-4af3-ad90-adce419f8b25\") " pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:48.507425 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:48.507397    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fw8v"
Sep 08 15:18:48.518368 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:48.518338    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77f885d_cd3e_4af3_ad90_adce419f8b25.slice/crio-9092595c10085dfc15a97a54f40b8b768806b00990f8efb5fc279b9597a87bd7 WatchSource:0}: Error finding container 9092595c10085dfc15a97a54f40b8b768806b00990f8efb5fc279b9597a87bd7: Status 404 returned error can't find the container with id 9092595c10085dfc15a97a54f40b8b768806b00990f8efb5fc279b9597a87bd7
Sep 08 15:18:48.763241 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:48.763211    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fw8v" event={"ID":"d77f885d-cd3e-4af3-ad90-adce419f8b25","Type":"ContainerStarted","Data":"9092595c10085dfc15a97a54f40b8b768806b00990f8efb5fc279b9597a87bd7"}
Sep 08 15:18:50.772043 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:50.772006    2575 generic.go:356] "Generic (PLEG): container finished" podID="d77f885d-cd3e-4af3-ad90-adce419f8b25" containerID="8f1dc5157a70aa389748812981cae586c3300385a605fc6bcfc285267621abe2" exitCode=0
Sep 08 15:18:50.772043 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:50.772045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fw8v" event={"ID":"d77f885d-cd3e-4af3-ad90-adce419f8b25","Type":"ContainerDied","Data":"8f1dc5157a70aa389748812981cae586c3300385a605fc6bcfc285267621abe2"}
Sep 08 15:18:51.777727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:51.777691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fw8v" event={"ID":"d77f885d-cd3e-4af3-ad90-adce419f8b25","Type":"ContainerStarted","Data":"6fb6917a08917280d6cd10a8e72d342d2b7d8556f5bef0299a18648047ab8215"}
Sep 08 15:18:51.778140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:51.777743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fw8v" event={"ID":"d77f885d-cd3e-4af3-ad90-adce419f8b25","Type":"ContainerStarted","Data":"8a321bd20d997df2a730d3d22a9fb2a5c0e2f7990acb8d3d8c75a53a8da6874d"}
Sep 08 15:18:51.803709 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:51.803651    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8fw8v" podStartSLOduration=3.514256572 podStartE2EDuration="4.803635929s" totalImagesPullingTime="1.222151162s" totalInitContainerRuntime="67.228195ms" isStatefulPod=true podCreationTimestamp="2026-09-08 15:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:51.801183392 +0000 UTC m=+80.876475697" watchObservedRunningTime="2026-09-08 15:18:51.803635929 +0000 UTC m=+80.878928219"
Sep 08 15:18:52.322345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.322309    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"]
Sep 08 15:18:52.327958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.327940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.331652 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.331626    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6fz2c\""
Sep 08 15:18:52.331820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.331804    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 15:18:52.336612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.336590    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"]
Sep 08 15:18:52.422274 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.422188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3340d218-0e75-47b1-92b2-b257f24b51ce-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6qhkd\" (UID: \"3340d218-0e75-47b1-92b2-b257f24b51ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.422274 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.422218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7cr9\" (UniqueName: \"kubernetes.io/projected/3340d218-0e75-47b1-92b2-b257f24b51ce-kube-api-access-t7cr9\") pod \"monitoring-plugin-85bbbbbf4-6qhkd\" (UID: \"3340d218-0e75-47b1-92b2-b257f24b51ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.522745 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.522720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3340d218-0e75-47b1-92b2-b257f24b51ce-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6qhkd\" (UID: \"3340d218-0e75-47b1-92b2-b257f24b51ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.522826 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.522750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7cr9\" (UniqueName: \"kubernetes.io/projected/3340d218-0e75-47b1-92b2-b257f24b51ce-kube-api-access-t7cr9\") pod \"monitoring-plugin-85bbbbbf4-6qhkd\" (UID: \"3340d218-0e75-47b1-92b2-b257f24b51ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.525183 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.525160    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3340d218-0e75-47b1-92b2-b257f24b51ce-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6qhkd\" (UID: \"3340d218-0e75-47b1-92b2-b257f24b51ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.531334 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.531308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7cr9\" (UniqueName: \"kubernetes.io/projected/3340d218-0e75-47b1-92b2-b257f24b51ce-kube-api-access-t7cr9\") pod \"monitoring-plugin-85bbbbbf4-6qhkd\" (UID: \"3340d218-0e75-47b1-92b2-b257f24b51ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.638826 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.638803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:52.908345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:52.908317    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"]
Sep 08 15:18:52.910786 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:52.910760    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3340d218_0e75_47b1_92b2_b257f24b51ce.slice/crio-8f40e0feb5e63cf471a057b321471664c453c5be778f49984024c470b4f33b45 WatchSource:0}: Error finding container 8f40e0feb5e63cf471a057b321471664c453c5be778f49984024c470b4f33b45: Status 404 returned error can't find the container with id 8f40e0feb5e63cf471a057b321471664c453c5be778f49984024c470b4f33b45
Sep 08 15:18:53.786805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.786768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd" event={"ID":"3340d218-0e75-47b1-92b2-b257f24b51ce","Type":"ContainerStarted","Data":"8f40e0feb5e63cf471a057b321471664c453c5be778f49984024c470b4f33b45"}
Sep 08 15:18:53.830493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.830455    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:18:53.837055 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.837031    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.840313 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.840291    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 15:18:53.840313 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.840303    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 15:18:53.840956 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.840936    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 15:18:53.841613 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.841590    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 15:18:53.841672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.841599    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1gp36icjrh1un\""
Sep 08 15:18:53.841773 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.841734    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 15:18:53.841773 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.841742    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 15:18:53.841773 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.841745    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 15:18:53.841977 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.841923    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 15:18:53.842031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.842017    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tdlq7\""
Sep 08 15:18:53.842078 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.842062    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 15:18:53.842254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.842239    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 15:18:53.843717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.843697    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 15:18:53.843717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.843705    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 15:18:53.843901 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.843721    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 15:18:53.847696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.847676    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 15:18:53.850481 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.850458    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 15:18:53.853450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.853429    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:18:53.931156 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-config\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqntp\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-kube-api-access-zqntp\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931586    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931627    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:53.931736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:53.931649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-config\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.032696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqntp\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-kube-api-access-zqntp\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.032982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.033010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033682 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.033034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033682 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.033573    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.033790 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.033740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.036314 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.034719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.036993 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.036758    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.037422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.037387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.037662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.037635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-config\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.037992 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.037967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.038136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.037983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.038232 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.038138    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.038322 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.038271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.038636 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.038611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.039027 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.038988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.039210 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.039152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.039424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.039399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.039788 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.039742    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.039928 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.039795    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.039982 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.039937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.040093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.040072    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.040252 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.040227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.046299 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.046268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqntp\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-kube-api-access-zqntp\") pod \"prometheus-k8s-0\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.149429 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.149396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:18:54.438575 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.438553    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:18:54.440153 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:18:54.440118    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e6958a_eac7_4cad_8291_42e4b5d822f3.slice/crio-20b3f7d8b7766e7c6fd3ceac0341f10166cfaf79fac5fee896e5a9301581af43 WatchSource:0}: Error finding container 20b3f7d8b7766e7c6fd3ceac0341f10166cfaf79fac5fee896e5a9301581af43: Status 404 returned error can't find the container with id 20b3f7d8b7766e7c6fd3ceac0341f10166cfaf79fac5fee896e5a9301581af43
Sep 08 15:18:54.792232 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:54.792141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"20b3f7d8b7766e7c6fd3ceac0341f10166cfaf79fac5fee896e5a9301581af43"}
Sep 08 15:18:55.755749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.755718    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v9cbd"
Sep 08 15:18:55.797590 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.797558    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd" event={"ID":"3340d218-0e75-47b1-92b2-b257f24b51ce","Type":"ContainerStarted","Data":"d36177e2c36f6e18c087ccdee2ab507aaf818f2dc9f5b532d6af88a7150fa0c5"}
Sep 08 15:18:55.797720 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.797695    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:55.800068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.800040    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18" exitCode=0
Sep 08 15:18:55.800168 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.800093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"}
Sep 08 15:18:55.802827 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.802805    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd"
Sep 08 15:18:55.821456 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:55.821417    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6qhkd" podStartSLOduration=1.062378255 podStartE2EDuration="3.821402663s" totalImagesPullingTime="2.759024408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:55.819898582 +0000 UTC m=+84.895190900" watchObservedRunningTime="2026-09-08 15:18:55.821402663 +0000 UTC m=+84.896694956"
Sep 08 15:18:59.827642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:18:59.827609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"}
Sep 08 15:19:00.834328 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:00.834292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"}
Sep 08 15:19:02.845931 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:02.845885    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"}
Sep 08 15:19:02.846324 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:02.846013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"}
Sep 08 15:19:02.846324 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:02.846059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"}
Sep 08 15:19:02.846324 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:02.846105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerStarted","Data":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"}
Sep 08 15:19:02.881416 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:02.881333    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7518701759999997 podStartE2EDuration="9.88130835s" totalImagesPullingTime="7.10246815s" totalInitContainerRuntime="26.970024ms" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:02.88000517 +0000 UTC m=+91.955297475" watchObservedRunningTime="2026-09-08 15:19:02.88130835 +0000 UTC m=+91.956600645"
Sep 08 15:19:04.149646 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:04.149603    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:19:16.755359 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:16.755327    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mbkb4"
Sep 08 15:19:54.149562 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:54.149520    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:19:54.172226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:54.172194    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:19:55.056863 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:19:55.056825    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:12.280079 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280037    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:20:12.280581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280406    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="prometheus" containerID="cri-o://6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37" gracePeriod=600
Sep 08 15:20:12.280581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280477    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy" containerID="cri-o://27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f" gracePeriod=600
Sep 08 15:20:12.280581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280517    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-web" containerID="cri-o://9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5" gracePeriod=600
Sep 08 15:20:12.280756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280486    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="config-reloader" containerID="cri-o://9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b" gracePeriod=600
Sep 08 15:20:12.280756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280486    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="thanos-sidecar" containerID="cri-o://f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a" gracePeriod=600
Sep 08 15:20:12.280756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.280517    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837" gracePeriod=600
Sep 08 15:20:12.512611 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.512589    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:12.588490 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588437    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-kubelet-serving-ca-bundle\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588490 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588469    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-2\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-db\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588515    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-tls\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588530    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-config\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588560    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588778    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-config-out\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.588934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588826    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588825    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.589045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588890    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.589045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.588873    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-trusted-ca-bundle\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589005    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-metrics-client-certs\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589244 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589045    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-1\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589647 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589288    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.589647 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-metrics-client-ca\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589660    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-tls-assets\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589691    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-grpc-tls\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqntp\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-kube-api-access-zqntp\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589746    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-kube-rbac-proxy\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589779    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-serving-certs-ca-bundle\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589804    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-0\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589837    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-web-config\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.589907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589894    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-thanos-prometheus-http-client-file\") pod \"01e6958a-eac7-4cad-8291-42e4b5d822f3\" (UID: \"01e6958a-eac7-4cad-8291-42e4b5d822f3\") "
Sep 08 15:20:12.590276 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.590126    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.590276 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.590144    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.590276 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.590159    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.592141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.589654    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.592141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.590013    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.592141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.590087    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:20:12.592141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.590741    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.592141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.591736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.592141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.592106    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:12.592665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.592556    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.593174 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.593136    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-config" (OuterVolumeSpecName: "config") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.593174 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.593165    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-config-out" (OuterVolumeSpecName: "config-out") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:20:12.593327 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.593191    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.593995 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.593965    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:12.594204 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.594187    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.594600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.594583    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-kube-api-access-zqntp" (OuterVolumeSpecName: "kube-api-access-zqntp") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "kube-api-access-zqntp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:12.594920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.594904    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.595083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.595058    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.595083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.595074    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.604942 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.604923    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-web-config" (OuterVolumeSpecName: "web-config") pod "01e6958a-eac7-4cad-8291-42e4b5d822f3" (UID: "01e6958a-eac7-4cad-8291-42e4b5d822f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:12.691217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691188    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691216    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-metrics-client-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691226    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-tls-assets\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691234    2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-grpc-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691242    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqntp\" (UniqueName: \"kubernetes.io/projected/01e6958a-eac7-4cad-8291-42e4b5d822f3-kube-api-access-zqntp\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691250    2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-kube-rbac-proxy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691258    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691276    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691284    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-web-config\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691293    2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691300    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-prometheus-k8s-db\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691308    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691316    2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-config\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691324    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691333    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01e6958a-eac7-4cad-8291-42e4b5d822f3-config-out\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691342    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:12.691364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:12.691350    2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01e6958a-eac7-4cad-8291-42e4b5d822f3-secret-metrics-client-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:20:13.106554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106526    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837" exitCode=0
Sep 08 15:20:13.106554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106549    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f" exitCode=0
Sep 08 15:20:13.106554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106555    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5" exitCode=0
Sep 08 15:20:13.106740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106561    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a" exitCode=0
Sep 08 15:20:13.106740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106567    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b" exitCode=0
Sep 08 15:20:13.106740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106572    2575 generic.go:356] "Generic (PLEG): container finished" podID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37" exitCode=0
Sep 08 15:20:13.106740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106615    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.106740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"}
Sep 08 15:20:13.106740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106714    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"}
Sep 08 15:20:13.106950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"}
Sep 08 15:20:13.106950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106780    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.106950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"}
Sep 08 15:20:13.106950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"}
Sep 08 15:20:13.106950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106915    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"}
Sep 08 15:20:13.107178 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.106955    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01e6958a-eac7-4cad-8291-42e4b5d822f3","Type":"ContainerDied","Data":"20b3f7d8b7766e7c6fd3ceac0341f10166cfaf79fac5fee896e5a9301581af43"}
Sep 08 15:20:13.116036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.116012    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.123536 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.123516    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.130415 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.130396    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.134251 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.134228    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:20:13.137946 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.137922    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.140645 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.140620    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:20:13.144919 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.144898    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.151822 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.151807    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.158484 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.158467    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.158787 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.158768    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.158835 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.158795    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"} err="failed to get container status \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist"
Sep 08 15:20:13.158835 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.158826    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.159142 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.159095    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.159185 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159147    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"} err="failed to get container status \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist"
Sep 08 15:20:13.159185 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159163    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.159401 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.159382    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.159465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159410    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"} err="failed to get container status \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist"
Sep 08 15:20:13.159465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159432    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.159681 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.159665    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.159717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159685    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"} err="failed to get container status \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist"
Sep 08 15:20:13.159717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159698    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.159923 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.159905    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.159998 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159928    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"} err="failed to get container status \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist"
Sep 08 15:20:13.159998 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.159949    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.160188 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.160172    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.160230 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160191    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"} err="failed to get container status \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist"
Sep 08 15:20:13.160230 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160205    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.160424 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:20:13.160410    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.160465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160429    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"} err="failed to get container status \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist"
Sep 08 15:20:13.160465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160441    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.160641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160624    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"} err="failed to get container status \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist"
Sep 08 15:20:13.160684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160641    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.160868 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160832    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"} err="failed to get container status \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist"
Sep 08 15:20:13.160916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.160870    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.161140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161119    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"} err="failed to get container status \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist"
Sep 08 15:20:13.161187 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161143    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.161396 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161376    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"} err="failed to get container status \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist"
Sep 08 15:20:13.161440 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161397    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.161649 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161631    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"} err="failed to get container status \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist"
Sep 08 15:20:13.161649 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161647    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.161947 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161855    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"} err="failed to get container status \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist"
Sep 08 15:20:13.161947 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.161870    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.162123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162105    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"} err="failed to get container status \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist"
Sep 08 15:20:13.162123 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162122    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.162343 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162324    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"} err="failed to get container status \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist"
Sep 08 15:20:13.162385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162344    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.162576 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162560    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"} err="failed to get container status \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist"
Sep 08 15:20:13.162576 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162576    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.162798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162782    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"} err="failed to get container status \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist"
Sep 08 15:20:13.162859 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.162798    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.163031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163015    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"} err="failed to get container status \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist"
Sep 08 15:20:13.163031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163030    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.163247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163231    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"} err="failed to get container status \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist"
Sep 08 15:20:13.163286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163246    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.163435 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163420    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"} err="failed to get container status \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist"
Sep 08 15:20:13.163481 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163434    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.163654 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163634    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"} err="failed to get container status \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist"
Sep 08 15:20:13.163654 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163653    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.163874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163838    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"} err="failed to get container status \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist"
Sep 08 15:20:13.163915 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.163876    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.164068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164052    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"} err="failed to get container status \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist"
Sep 08 15:20:13.164107 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164067    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.164291 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164273    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"} err="failed to get container status \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist"
Sep 08 15:20:13.164345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164293    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.164522 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164500    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"} err="failed to get container status \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist"
Sep 08 15:20:13.164568 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164524    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.164710 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164693    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"} err="failed to get container status \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist"
Sep 08 15:20:13.164749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164710    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.164933 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164917    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"} err="failed to get container status \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist"
Sep 08 15:20:13.164999 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.164932    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.165163 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165148    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"} err="failed to get container status \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist"
Sep 08 15:20:13.165208 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165163    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.165394 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165377    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"} err="failed to get container status \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist"
Sep 08 15:20:13.165440 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165394    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.165577 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165560    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"} err="failed to get container status \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist"
Sep 08 15:20:13.165577 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165577    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.165810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165794    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"} err="failed to get container status \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist"
Sep 08 15:20:13.165884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.165811    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.166039 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166021    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"} err="failed to get container status \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist"
Sep 08 15:20:13.166079 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166039    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.166215 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166200    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"} err="failed to get container status \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist"
Sep 08 15:20:13.166255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166215    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.166446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166430    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"} err="failed to get container status \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist"
Sep 08 15:20:13.166446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166445    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.166617 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166597    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"} err="failed to get container status \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist"
Sep 08 15:20:13.166662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166619    2575 scope.go:117] "RemoveContainer" containerID="46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"
Sep 08 15:20:13.166805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166782    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837"} err="failed to get container status \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": rpc error: code = NotFound desc = could not find container \"46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837\": container with ID starting with 46116dba983cefb961050544756f957cb8aa1f4f594f8f299165832846ac7837 not found: ID does not exist"
Sep 08 15:20:13.166805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.166804    2575 scope.go:117] "RemoveContainer" containerID="27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"
Sep 08 15:20:13.167018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167001    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f"} err="failed to get container status \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": rpc error: code = NotFound desc = could not find container \"27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f\": container with ID starting with 27eba16c801c562651908b4563f1b160ae1683f6d980ea5ac39061846b12974f not found: ID does not exist"
Sep 08 15:20:13.167066 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167020    2575 scope.go:117] "RemoveContainer" containerID="9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"
Sep 08 15:20:13.167260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167241    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5"} err="failed to get container status \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": rpc error: code = NotFound desc = could not find container \"9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5\": container with ID starting with 9b411ab4779297ef1353943c8a80e4c103b3674d97878682517565ff8ff6cbf5 not found: ID does not exist"
Sep 08 15:20:13.167326 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167262    2575 scope.go:117] "RemoveContainer" containerID="f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"
Sep 08 15:20:13.167484 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167465    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a"} err="failed to get container status \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": rpc error: code = NotFound desc = could not find container \"f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a\": container with ID starting with f7d7c060bd18a8cf0ce186a71ec84c34d3e0a7997e2e72e95820eab289736b3a not found: ID does not exist"
Sep 08 15:20:13.167529 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167485    2575 scope.go:117] "RemoveContainer" containerID="9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"
Sep 08 15:20:13.167710 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167688    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b"} err="failed to get container status \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": rpc error: code = NotFound desc = could not find container \"9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b\": container with ID starting with 9b2047e580395e1817f5335ab57e54442b7d1d565cb9eee32db458383078157b not found: ID does not exist"
Sep 08 15:20:13.167750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167711    2575 scope.go:117] "RemoveContainer" containerID="6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"
Sep 08 15:20:13.167938 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167920    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37"} err="failed to get container status \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": rpc error: code = NotFound desc = could not find container \"6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37\": container with ID starting with 6fbd3d71fc3698ce0240487628dd5d8ae6a9413da8656171d020e6ba9fe48f37 not found: ID does not exist"
Sep 08 15:20:13.167989 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.167939    2575 scope.go:117] "RemoveContainer" containerID="c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"
Sep 08 15:20:13.168157 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.168139    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18"} err="failed to get container status \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": rpc error: code = NotFound desc = could not find container \"c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18\": container with ID starting with c174c79c9c5fc4c45277b8b3a9bb479384ea1a064be9b87385d240497d293f18 not found: ID does not exist"
Sep 08 15:20:13.190718 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190698    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:20:13.190967 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190954    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="thanos-sidecar"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190969    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="thanos-sidecar"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190977    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190982    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190990    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="config-reloader"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.190996    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="config-reloader"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191001    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-web"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191007    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-web"
Sep 08 15:20:13.191026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191024    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="init-config-reloader"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191033    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="init-config-reloader"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191038    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-thanos"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191046    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-thanos"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191060    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="prometheus"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191065    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="prometheus"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191110    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-thanos"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191118    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="prometheus"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191124    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="thanos-sidecar"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191129    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy-web"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191134    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="kube-rbac-proxy"
Sep 08 15:20:13.191246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.191139    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" containerName="config-reloader"
Sep 08 15:20:13.196329 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.196306    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.201361 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.201341    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 15:20:13.201555 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.201542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 15:20:13.202218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.202062    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 15:20:13.205384 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205364    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 15:20:13.205384 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205390    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 15:20:13.205550 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205444    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 15:20:13.205550 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205471    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 15:20:13.205550 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205489    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 15:20:13.205550 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205521    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1gp36icjrh1un\""
Sep 08 15:20:13.205825 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205737    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 15:20:13.205825 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205786    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 15:20:13.205825 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205793    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 15:20:13.205825 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205817    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 15:20:13.206039 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.205956    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tdlq7\""
Sep 08 15:20:13.209241 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.209226    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 15:20:13.212803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.212780    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 15:20:13.219209 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.219193    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 15:20:13.227721 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.227705    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:20:13.295695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.295695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694pk\" (UniqueName: \"kubernetes.io/projected/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-kube-api-access-694pk\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-config\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.295985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.296356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.296148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.396974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694pk\" (UniqueName: \"kubernetes.io/projected/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-kube-api-access-694pk\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-config\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.401226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.397623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.400706    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.401559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.401934    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.402308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.402583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.402457    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.403761 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.403709    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.404702 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.404670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.404909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.404883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.405300 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.405250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.405592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.405536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.405937 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.405785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.405937 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.405788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.406082 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.406043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.407130 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.407016    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.407130 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.407088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.407277 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.407176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.407277 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.407249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-config\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.407811 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.407790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.407978 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.407959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.412938 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.412914    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-694pk\" (UniqueName: \"kubernetes.io/projected/43f3f747-97a9-4992-8e4a-cc44fd93f4d8-kube-api-access-694pk\") pod \"prometheus-k8s-0\" (UID: \"43f3f747-97a9-4992-8e4a-cc44fd93f4d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.439906 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.439886    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e6958a-eac7-4cad-8291-42e4b5d822f3" path="/var/lib/kubelet/pods/01e6958a-eac7-4cad-8291-42e4b5d822f3/volumes"
Sep 08 15:20:13.504530 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.504497    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:20:13.784692 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:13.784634    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 15:20:13.786403 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:20:13.786371    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f3f747_97a9_4992_8e4a_cc44fd93f4d8.slice/crio-508311191bea6f5acdfc2b3e5e7f2ab318cb7c94daa93fcb07824cde2ee76cf8 WatchSource:0}: Error finding container 508311191bea6f5acdfc2b3e5e7f2ab318cb7c94daa93fcb07824cde2ee76cf8: Status 404 returned error can't find the container with id 508311191bea6f5acdfc2b3e5e7f2ab318cb7c94daa93fcb07824cde2ee76cf8
Sep 08 15:20:14.113458 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:14.113387    2575 generic.go:356] "Generic (PLEG): container finished" podID="43f3f747-97a9-4992-8e4a-cc44fd93f4d8" containerID="d82eb32ec1fac7a09888c11b15c82053d044b93dfbd87189a2a63acf61697cb5" exitCode=0
Sep 08 15:20:14.113458 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:14.113440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerDied","Data":"d82eb32ec1fac7a09888c11b15c82053d044b93dfbd87189a2a63acf61697cb5"}
Sep 08 15:20:14.113593 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:14.113507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"508311191bea6f5acdfc2b3e5e7f2ab318cb7c94daa93fcb07824cde2ee76cf8"}
Sep 08 15:20:15.121891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.121839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"2a35d03adc4e153c957317bd317bd517b0130bcf4778debeb8f91d7d2b29bac2"}
Sep 08 15:20:15.122369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.121933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"ed5e772f87fc3c0561e03e4391e07baa248dfc66bfe7e103d7a6c08ad5b4b5e0"}
Sep 08 15:20:15.122369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.121975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"9af8d740d216aca42e7a7c652fa962d1e09cc6e0f92d1fd432db198c6b592d52"}
Sep 08 15:20:15.122369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.122027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"eb2d89e3f22e1ccafc5ce6b51c8a01f721ddab655c9f0207fa5fa00a5a5fb503"}
Sep 08 15:20:15.122369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.122068    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"9cbdd2be844ce18dab2b32bb09412cf832f3e26756cc9ee7ce0c888d71f1fc39"}
Sep 08 15:20:15.122369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.122114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43f3f747-97a9-4992-8e4a-cc44fd93f4d8","Type":"ContainerStarted","Data":"580c9d2406551b7557618ce705ff1e824b8ed54ef7df7a79ee18df4e68b64c74"}
Sep 08 15:20:15.156289 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:15.156218    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.137476268 podStartE2EDuration="2.156202648s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.72638ms" isStatefulPod=false podCreationTimestamp="2026-09-08 15:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:20:15.153917235 +0000 UTC m=+164.229209549" watchObservedRunningTime="2026-09-08 15:20:15.156202648 +0000 UTC m=+164.231494941"
Sep 08 15:20:18.505251 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:20:18.505202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:21:13.505613 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:21:13.505567    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:21:13.523369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:21:13.523331    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:21:14.356593 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:21:14.356567    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 15:22:31.320393 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:31.320361    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 15:22:50.459919 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.459889    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"]
Sep 08 15:22:50.463185 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.463160    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.466102 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.466077    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 15:22:50.471595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.471573    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 15:22:50.471699 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.471614    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wv984\""
Sep 08 15:22:50.471699 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.471628    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 15:22:50.471916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.471900    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 15:22:50.471966 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.471933    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:22:50.493090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.493071    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"]
Sep 08 15:22:50.564121 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.564099    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/202837a5-dc72-49fa-b759-1a3cb0b64eed-manager-config\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.564220 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.564135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tdt\" (UniqueName: \"kubernetes.io/projected/202837a5-dc72-49fa-b759-1a3cb0b64eed-kube-api-access-j8tdt\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.564220 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.564157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/202837a5-dc72-49fa-b759-1a3cb0b64eed-cert\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.564220 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.564177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/202837a5-dc72-49fa-b759-1a3cb0b64eed-metrics-cert\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.637304 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.637280    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"]
Sep 08 15:22:50.640548 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.640531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.646834 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.646814    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 15:22:50.646939 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.646839    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4lc5s\""
Sep 08 15:22:50.647022 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.647007    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 15:22:50.647408 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.647394    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 15:22:50.647462 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.647448    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 15:22:50.664403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.664381    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"]
Sep 08 15:22:50.664630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.664582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/202837a5-dc72-49fa-b759-1a3cb0b64eed-manager-config\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.664630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.664623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tdt\" (UniqueName: \"kubernetes.io/projected/202837a5-dc72-49fa-b759-1a3cb0b64eed-kube-api-access-j8tdt\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.664764 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.664652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/202837a5-dc72-49fa-b759-1a3cb0b64eed-cert\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.664764 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.664685    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/202837a5-dc72-49fa-b759-1a3cb0b64eed-metrics-cert\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.665306 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.665289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/202837a5-dc72-49fa-b759-1a3cb0b64eed-manager-config\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.667203 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.667182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/202837a5-dc72-49fa-b759-1a3cb0b64eed-metrics-cert\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.667415 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.667399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/202837a5-dc72-49fa-b759-1a3cb0b64eed-cert\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.680218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.680191    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tdt\" (UniqueName: \"kubernetes.io/projected/202837a5-dc72-49fa-b759-1a3cb0b64eed-kube-api-access-j8tdt\") pod \"lws-controller-manager-5bc9d6d9f9-m8fwk\" (UID: \"202837a5-dc72-49fa-b759-1a3cb0b64eed\") " pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.765984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.765930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-manifests\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.766064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.765983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.766064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.766021    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-tmp\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.766064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.766044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksrq\" (UniqueName: \"kubernetes.io/projected/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-kube-api-access-hksrq\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.766164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.766124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.772939 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.772921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:50.867190 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.867158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksrq\" (UniqueName: \"kubernetes.io/projected/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-kube-api-access-hksrq\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.867303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.867206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.867303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.867245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-manifests\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.867303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.867263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.867303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.867296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-tmp\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.867739 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.867714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-manifests\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.869380 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.869354    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-tmp\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.869663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.869644    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.869702 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.869675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.892940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.892920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksrq\" (UniqueName: \"kubernetes.io/projected/8ed1657f-7a61-4f27-a5ca-6671c615f9cc-kube-api-access-hksrq\") pod \"opendatahub-operator-controller-manager-847d47f6c8-w8fgf\" (UID: \"8ed1657f-7a61-4f27-a5ca-6671c615f9cc\") " pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:50.949469 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:50.949447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:51.055839 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:51.055812    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"]
Sep 08 15:22:51.056480 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:22:51.056456    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202837a5_dc72_49fa_b759_1a3cb0b64eed.slice/crio-f9d008d5fab8202bca9fd71d02e3f3a34d3d2ca20d3d12019168919527ca41f4 WatchSource:0}: Error finding container f9d008d5fab8202bca9fd71d02e3f3a34d3d2ca20d3d12019168919527ca41f4: Status 404 returned error can't find the container with id f9d008d5fab8202bca9fd71d02e3f3a34d3d2ca20d3d12019168919527ca41f4
Sep 08 15:22:51.058348 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:51.058330    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:22:51.226147 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:51.226048    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"]
Sep 08 15:22:51.228631 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:22:51.228602    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed1657f_7a61_4f27_a5ca_6671c615f9cc.slice/crio-ea72aa776e9b1e7b955b524945963d000a4683fe9b4f2cb8e8421911d7a53f2f WatchSource:0}: Error finding container ea72aa776e9b1e7b955b524945963d000a4683fe9b4f2cb8e8421911d7a53f2f: Status 404 returned error can't find the container with id ea72aa776e9b1e7b955b524945963d000a4683fe9b4f2cb8e8421911d7a53f2f
Sep 08 15:22:51.689347 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:51.689318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf" event={"ID":"8ed1657f-7a61-4f27-a5ca-6671c615f9cc","Type":"ContainerStarted","Data":"ea72aa776e9b1e7b955b524945963d000a4683fe9b4f2cb8e8421911d7a53f2f"}
Sep 08 15:22:51.690518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:51.690497    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk" event={"ID":"202837a5-dc72-49fa-b759-1a3cb0b64eed","Type":"ContainerStarted","Data":"f9d008d5fab8202bca9fd71d02e3f3a34d3d2ca20d3d12019168919527ca41f4"}
Sep 08 15:22:54.705701 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:54.705645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk" event={"ID":"202837a5-dc72-49fa-b759-1a3cb0b64eed","Type":"ContainerStarted","Data":"206dc99b07e43cb420169d12a0fc7e5181343bc63376b1dea129252e9fa0d5cc"}
Sep 08 15:22:54.706259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:54.706235    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:22:54.740555 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:54.740508    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk" podStartSLOduration=2.11752484 podStartE2EDuration="4.740487463s" totalImagesPullingTime="2.622962623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:22:54.740130313 +0000 UTC m=+323.815422626" watchObservedRunningTime="2026-09-08 15:22:54.740487463 +0000 UTC m=+323.815779757"
Sep 08 15:22:56.714971 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:56.714937    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf" event={"ID":"8ed1657f-7a61-4f27-a5ca-6671c615f9cc","Type":"ContainerStarted","Data":"82e30da2fe1d6d7ac46924d9e94846408e7394e08f15f5a0f43243c06bb7f092"}
Sep 08 15:22:57.721590 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:57.721546    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ed1657f-7a61-4f27-a5ca-6671c615f9cc" containerID="82e30da2fe1d6d7ac46924d9e94846408e7394e08f15f5a0f43243c06bb7f092" exitCode=0
Sep 08 15:22:57.721999 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:57.721646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf" event={"ID":"8ed1657f-7a61-4f27-a5ca-6671c615f9cc","Type":"ContainerDied","Data":"82e30da2fe1d6d7ac46924d9e94846408e7394e08f15f5a0f43243c06bb7f092"}
Sep 08 15:22:58.730188 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:58.730145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf" event={"ID":"8ed1657f-7a61-4f27-a5ca-6671c615f9cc","Type":"ContainerStarted","Data":"4efadbfbd162aafef9487d100c1854b3190a38da6fb9c46cfa1498dfaf13f0b2"}
Sep 08 15:22:58.730604 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:58.730265    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:22:58.755958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:22:58.755909    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf" podStartSLOduration=2.5123188020000002 podStartE2EDuration="8.75589477s" totalImagesPullingTime="5.199529668s" totalInitContainerRuntime="1.0440463s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:22:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:22:58.754113859 +0000 UTC m=+327.829406163" watchObservedRunningTime="2026-09-08 15:22:58.75589477 +0000 UTC m=+327.831187063"
Sep 08 15:23:06.717403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:06.717367    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bc9d6d9f9-m8fwk"
Sep 08 15:23:09.737012 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:09.736973    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-847d47f6c8-w8fgf"
Sep 08 15:23:12.646700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.646672    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"]
Sep 08 15:23:12.696543 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.696514    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"]
Sep 08 15:23:12.696677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.696621    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.699774 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.699744    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 15:23:12.699774 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.699755    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 15:23:12.699984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.699756    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 15:23:12.699984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.699789    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sfjkh\""
Sep 08 15:23:12.700862 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.700825    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 15:23:12.830595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.830571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04111a4c-5278-49c3-a692-a3412baf5e08-tls-certs\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.830757 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.830607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4zx\" (UniqueName: \"kubernetes.io/projected/04111a4c-5278-49c3-a692-a3412baf5e08-kube-api-access-kf4zx\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.830757 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.830628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04111a4c-5278-49c3-a692-a3412baf5e08-tmp\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.931979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.931921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04111a4c-5278-49c3-a692-a3412baf5e08-tls-certs\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.931979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.931958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4zx\" (UniqueName: \"kubernetes.io/projected/04111a4c-5278-49c3-a692-a3412baf5e08-kube-api-access-kf4zx\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.931979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.931977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04111a4c-5278-49c3-a692-a3412baf5e08-tmp\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.934553 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.934522    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04111a4c-5278-49c3-a692-a3412baf5e08-tmp\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.934656 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.934569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04111a4c-5278-49c3-a692-a3412baf5e08-tls-certs\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:12.941383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:12.941357    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4zx\" (UniqueName: \"kubernetes.io/projected/04111a4c-5278-49c3-a692-a3412baf5e08-kube-api-access-kf4zx\") pod \"kube-auth-proxy-7d44f46d57-7dn5g\" (UID: \"04111a4c-5278-49c3-a692-a3412baf5e08\") " pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:13.006097 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:13.006076    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"
Sep 08 15:23:13.315089 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:13.315049    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g"]
Sep 08 15:23:13.317716 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:23:13.317689    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04111a4c_5278_49c3_a692_a3412baf5e08.slice/crio-ba8889dfa1ddae9bdde66921a3cef24b443c499cc3e18429b8a27b5f3ada321b WatchSource:0}: Error finding container ba8889dfa1ddae9bdde66921a3cef24b443c499cc3e18429b8a27b5f3ada321b: Status 404 returned error can't find the container with id ba8889dfa1ddae9bdde66921a3cef24b443c499cc3e18429b8a27b5f3ada321b
Sep 08 15:23:13.793004 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:13.792967    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g" event={"ID":"04111a4c-5278-49c3-a692-a3412baf5e08","Type":"ContainerStarted","Data":"ba8889dfa1ddae9bdde66921a3cef24b443c499cc3e18429b8a27b5f3ada321b"}
Sep 08 15:23:16.807197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:16.807161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g" event={"ID":"04111a4c-5278-49c3-a692-a3412baf5e08","Type":"ContainerStarted","Data":"a5606a5a8c5dae24382495aa2c5c3fe7a3df14a275afcb7ff1c8b6e610947a1f"}
Sep 08 15:23:16.828219 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:23:16.828169    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d44f46d57-7dn5g" podStartSLOduration=2.201935885 podStartE2EDuration="4.828151327s" totalImagesPullingTime="2.626215442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:23:16.82687468 +0000 UTC m=+345.902166984" watchObservedRunningTime="2026-09-08 15:23:16.828151327 +0000 UTC m=+345.903443620"
Sep 08 15:24:49.449260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.449221    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"]
Sep 08 15:24:49.452653 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.452634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:24:49.456425 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.456404    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 15:24:49.457345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.457325    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5bk8c\""
Sep 08 15:24:49.457408 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.457376    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 15:24:49.464327 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.464301    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"]
Sep 08 15:24:49.621070 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.621038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskrv\" (UniqueName: \"kubernetes.io/projected/b3489a30-9144-44c5-b2e0-32232889c665-kube-api-access-gskrv\") pod \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" (UID: \"b3489a30-9144-44c5-b2e0-32232889c665\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:24:49.722061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.721998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskrv\" (UniqueName: \"kubernetes.io/projected/b3489a30-9144-44c5-b2e0-32232889c665-kube-api-access-gskrv\") pod \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" (UID: \"b3489a30-9144-44c5-b2e0-32232889c665\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:24:49.731583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.731547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskrv\" (UniqueName: \"kubernetes.io/projected/b3489a30-9144-44c5-b2e0-32232889c665-kube-api-access-gskrv\") pod \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" (UID: \"b3489a30-9144-44c5-b2e0-32232889c665\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:24:49.763771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:49.763741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:24:50.039281 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.039215    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"]
Sep 08 15:24:50.042703 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:24:50.042677    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3489a30_9144_44c5_b2e0_32232889c665.slice/crio-52a630470e92a68da027ce7002d9fa426db2ddf980878fcbeb9ca09051bd1012 WatchSource:0}: Error finding container 52a630470e92a68da027ce7002d9fa426db2ddf980878fcbeb9ca09051bd1012: Status 404 returned error can't find the container with id 52a630470e92a68da027ce7002d9fa426db2ddf980878fcbeb9ca09051bd1012
Sep 08 15:24:50.183797 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.183768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc" event={"ID":"b3489a30-9144-44c5-b2e0-32232889c665","Type":"ContainerStarted","Data":"52a630470e92a68da027ce7002d9fa426db2ddf980878fcbeb9ca09051bd1012"}
Sep 08 15:24:50.357327 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.357264    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"]
Sep 08 15:24:50.361671 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.361653    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.366537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.366514    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 15:24:50.366537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.366533    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2cn8b\""
Sep 08 15:24:50.366776 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.366535    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 15:24:50.374688 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.374671    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"]
Sep 08 15:24:50.529771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.529742    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e30b2636-1909-4242-81e0-2be93bd24e21-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.530101 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.529777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ds9\" (UniqueName: \"kubernetes.io/projected/e30b2636-1909-4242-81e0-2be93bd24e21-kube-api-access-n6ds9\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.530101 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.529803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e30b2636-1909-4242-81e0-2be93bd24e21-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.631294 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.631224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e30b2636-1909-4242-81e0-2be93bd24e21-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.631409 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.631292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ds9\" (UniqueName: \"kubernetes.io/projected/e30b2636-1909-4242-81e0-2be93bd24e21-kube-api-access-n6ds9\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.631409 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.631323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e30b2636-1909-4242-81e0-2be93bd24e21-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.632316 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.632293    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e30b2636-1909-4242-81e0-2be93bd24e21-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.634124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.634099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e30b2636-1909-4242-81e0-2be93bd24e21-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.643367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.643347    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ds9\" (UniqueName: \"kubernetes.io/projected/e30b2636-1909-4242-81e0-2be93bd24e21-kube-api-access-n6ds9\") pod \"kuadrant-console-plugin-598f86468b-zsh5k\" (UID: \"e30b2636-1909-4242-81e0-2be93bd24e21\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.671215 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.671196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"
Sep 08 15:24:50.943048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:50.942675    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k"]
Sep 08 15:24:50.945455 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:24:50.945423    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode30b2636_1909_4242_81e0_2be93bd24e21.slice/crio-9906273130da19722eb6738d57ebdc04ad2402e4137b01dccbc9c841b7cf1cfe WatchSource:0}: Error finding container 9906273130da19722eb6738d57ebdc04ad2402e4137b01dccbc9c841b7cf1cfe: Status 404 returned error can't find the container with id 9906273130da19722eb6738d57ebdc04ad2402e4137b01dccbc9c841b7cf1cfe
Sep 08 15:24:51.190170 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:51.190140    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k" event={"ID":"e30b2636-1909-4242-81e0-2be93bd24e21","Type":"ContainerStarted","Data":"9906273130da19722eb6738d57ebdc04ad2402e4137b01dccbc9c841b7cf1cfe"}
Sep 08 15:24:53.204022 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:53.203981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc" event={"ID":"b3489a30-9144-44c5-b2e0-32232889c665","Type":"ContainerStarted","Data":"a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714"}
Sep 08 15:24:53.204724 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:53.204664    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:24:53.245224 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:53.245119    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc" podStartSLOduration=1.29438748 podStartE2EDuration="4.245089002s" totalImagesPullingTime="2.950701522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:24:53.244276818 +0000 UTC m=+442.319569123" watchObservedRunningTime="2026-09-08 15:24:53.245089002 +0000 UTC m=+442.320381296"
Sep 08 15:24:57.229000 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:57.228958    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k" event={"ID":"e30b2636-1909-4242-81e0-2be93bd24e21","Type":"ContainerStarted","Data":"1e61ff2c1b74861966561802b6b25d4b0346d399c11ba90b77fb43e38ac23b1b"}
Sep 08 15:24:57.249392 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:24:57.249342    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsh5k" podStartSLOduration=1.439045302 podStartE2EDuration="7.249325484s" totalImagesPullingTime="5.810280182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:24:57.248250476 +0000 UTC m=+446.323542770" watchObservedRunningTime="2026-09-08 15:24:57.249325484 +0000 UTC m=+446.324617777"
Sep 08 15:25:01.954528 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:01.954483    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"]
Sep 08 15:25:01.954914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:01.954753    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc" podUID="b3489a30-9144-44c5-b2e0-32232889c665" containerName="manager" containerID="cri-o://a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714" gracePeriod=2
Sep 08 15:25:01.956770 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:01.956747    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:01.972579 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:01.972557    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"]
Sep 08 15:25:01.980875 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:01.980799    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b3489a30-9144-44c5-b2e0-32232889c665\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-wjrfc\": pods \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:01.983734 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:01.983707    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="b3489a30-9144-44c5-b2e0-32232889c665" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:02.185610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.185588    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:02.188222 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:02.188196    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="b3489a30-9144-44c5-b2e0-32232889c665" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:02.204883 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.204831    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gskrv\" (UniqueName: \"kubernetes.io/projected/b3489a30-9144-44c5-b2e0-32232889c665-kube-api-access-gskrv\") pod \"b3489a30-9144-44c5-b2e0-32232889c665\" (UID: \"b3489a30-9144-44c5-b2e0-32232889c665\") "
Sep 08 15:25:02.206884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.206864    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3489a30-9144-44c5-b2e0-32232889c665-kube-api-access-gskrv" (OuterVolumeSpecName: "kube-api-access-gskrv") pod "b3489a30-9144-44c5-b2e0-32232889c665" (UID: "b3489a30-9144-44c5-b2e0-32232889c665"). InnerVolumeSpecName "kube-api-access-gskrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:25:02.252236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.252213    2575 generic.go:356] "Generic (PLEG): container finished" podID="b3489a30-9144-44c5-b2e0-32232889c665" containerID="a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714" exitCode=0
Sep 08 15:25:02.252316 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.252254    2575 scope.go:117] "RemoveContainer" containerID="a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714"
Sep 08 15:25:02.252316 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.252256    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:02.255567 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:02.255543    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="b3489a30-9144-44c5-b2e0-32232889c665" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:02.260672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.260658    2575 scope.go:117] "RemoveContainer" containerID="a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714"
Sep 08 15:25:02.260960 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:02.260939    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714\": container with ID starting with a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714 not found: ID does not exist" containerID="a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714"
Sep 08 15:25:02.261015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.260971    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714"} err="failed to get container status \"a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714\": rpc error: code = NotFound desc = could not find container \"a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714\": container with ID starting with a32b58f3a0081ca0a3c23ad3549a13eda2e8c8b4aabf6be714e8a8252d839714 not found: ID does not exist"
Sep 08 15:25:02.262819 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:02.262798    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-wjrfc\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="b3489a30-9144-44c5-b2e0-32232889c665" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wjrfc"
Sep 08 15:25:02.305356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:02.305340    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gskrv\" (UniqueName: \"kubernetes.io/projected/b3489a30-9144-44c5-b2e0-32232889c665-kube-api-access-gskrv\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:25:03.440665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:03.440629    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3489a30-9144-44c5-b2e0-32232889c665" path="/var/lib/kubelet/pods/b3489a30-9144-44c5-b2e0-32232889c665/volumes"
Sep 08 15:25:31.279108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.279077    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"]
Sep 08 15:25:31.279537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.279348    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3489a30-9144-44c5-b2e0-32232889c665" containerName="manager"
Sep 08 15:25:31.279537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.279359    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3489a30-9144-44c5-b2e0-32232889c665" containerName="manager"
Sep 08 15:25:31.279537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.279409    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3489a30-9144-44c5-b2e0-32232889c665" containerName="manager"
Sep 08 15:25:31.296716 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.296692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.300041 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.300016    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 15:25:31.300249 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.300229    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rt4nj\""
Sep 08 15:25:31.303356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.303335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"]
Sep 08 15:25:31.422548 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422517    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422899 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7170ec93-6665-4421-8ba3-9d6747ad0146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.422899 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.422775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9pg\" (UniqueName: \"kubernetes.io/projected/7170ec93-6665-4421-8ba3-9d6747ad0146-kube-api-access-mh9pg\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.523700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.523820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.523820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.523820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7170ec93-6665-4421-8ba3-9d6747ad0146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.523820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9pg\" (UniqueName: \"kubernetes.io/projected/7170ec93-6665-4421-8ba3-9d6747ad0146-kube-api-access-mh9pg\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.523977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524216 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524216 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524216 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524440 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.524498 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.524481    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7170ec93-6665-4421-8ba3-9d6747ad0146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.526260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.526234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.526592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.526571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.533328 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.533278    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9pg\" (UniqueName: \"kubernetes.io/projected/7170ec93-6665-4421-8ba3-9d6747ad0146-kube-api-access-mh9pg\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.533726 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.533710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7170ec93-6665-4421-8ba3-9d6747ad0146-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6k5qw\" (UID: \"7170ec93-6665-4421-8ba3-9d6747ad0146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.607890 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.607864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:31.883447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:31.883417    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"]
Sep 08 15:25:31.885203 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:25:31.885173    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7170ec93_6665_4421_8ba3_9d6747ad0146.slice/crio-7222c96845519a815ecd3bd0111860a3381d7b9694aee57b16249af6cf6e9c2e WatchSource:0}: Error finding container 7222c96845519a815ecd3bd0111860a3381d7b9694aee57b16249af6cf6e9c2e: Status 404 returned error can't find the container with id 7222c96845519a815ecd3bd0111860a3381d7b9694aee57b16249af6cf6e9c2e
Sep 08 15:25:32.393410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:32.393375    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" event={"ID":"7170ec93-6665-4421-8ba3-9d6747ad0146","Type":"ContainerStarted","Data":"7222c96845519a815ecd3bd0111860a3381d7b9694aee57b16249af6cf6e9c2e"}
Sep 08 15:25:34.529056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:34.529013    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:25:34.529342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:34.529092    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:25:34.529342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:34.529121    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:25:35.408712 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:35.408674    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" event={"ID":"7170ec93-6665-4421-8ba3-9d6747ad0146","Type":"ContainerStarted","Data":"1e76df51f995c1b187339eb49ac7bc90612ce6ad7ae607e6fa635422a6eadbab"}
Sep 08 15:25:35.437361 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:35.437316    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podStartSLOduration=1.795894047 podStartE2EDuration="4.437300996s" totalImagesPullingTime="2.641406949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:35.434960795 +0000 UTC m=+484.510253104" watchObservedRunningTime="2026-09-08 15:25:35.437300996 +0000 UTC m=+484.512593288"
Sep 08 15:25:35.608906 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:35.608874    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:35.613526 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:35.613502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:36.413133 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:36.413097    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:36.414272 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:36.414244    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:25:44.150570 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.150522    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:25:44.154272 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.154246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.157261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.157235    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 15:25:44.165924 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.165896    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:25:44.205411 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.205384    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:25:44.217976 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.217955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8znj\" (UniqueName: \"kubernetes.io/projected/55cef3ff-3c94-4cac-af00-f8025af07dce-kube-api-access-r8znj\") pod \"limitador-limitador-85d9db4d5d-h5268\" (UID: \"55cef3ff-3c94-4cac-af00-f8025af07dce\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.218032 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.217992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55cef3ff-3c94-4cac-af00-f8025af07dce-config-file\") pod \"limitador-limitador-85d9db4d5d-h5268\" (UID: \"55cef3ff-3c94-4cac-af00-f8025af07dce\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.318452 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.318421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8znj\" (UniqueName: \"kubernetes.io/projected/55cef3ff-3c94-4cac-af00-f8025af07dce-kube-api-access-r8znj\") pod \"limitador-limitador-85d9db4d5d-h5268\" (UID: \"55cef3ff-3c94-4cac-af00-f8025af07dce\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.318541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.318485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55cef3ff-3c94-4cac-af00-f8025af07dce-config-file\") pod \"limitador-limitador-85d9db4d5d-h5268\" (UID: \"55cef3ff-3c94-4cac-af00-f8025af07dce\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.319069 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.319053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55cef3ff-3c94-4cac-af00-f8025af07dce-config-file\") pod \"limitador-limitador-85d9db4d5d-h5268\" (UID: \"55cef3ff-3c94-4cac-af00-f8025af07dce\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.327565 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.327547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8znj\" (UniqueName: \"kubernetes.io/projected/55cef3ff-3c94-4cac-af00-f8025af07dce-kube-api-access-r8znj\") pod \"limitador-limitador-85d9db4d5d-h5268\" (UID: \"55cef3ff-3c94-4cac-af00-f8025af07dce\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.469132 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.469072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:44.560413 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.560385    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-9gvl5"]
Sep 08 15:25:44.564682 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.564663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:44.567801 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.567783    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rvx67\""
Sep 08 15:25:44.575058 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.575038    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-9gvl5"]
Sep 08 15:25:44.721393 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.721323    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drz7c\" (UniqueName: \"kubernetes.io/projected/bbdbbb1c-1d5e-4ede-8061-ca5347835316-kube-api-access-drz7c\") pod \"authorino-b54df94b4-9gvl5\" (UID: \"bbdbbb1c-1d5e-4ede-8061-ca5347835316\") " pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:44.744924 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.744903    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:25:44.747110 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:25:44.747085    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cef3ff_3c94_4cac_af00_f8025af07dce.slice/crio-0f010278cf21cd1a930cee1044d59ba1a32b120febad5e0677c19d4cb532db73 WatchSource:0}: Error finding container 0f010278cf21cd1a930cee1044d59ba1a32b120febad5e0677c19d4cb532db73: Status 404 returned error can't find the container with id 0f010278cf21cd1a930cee1044d59ba1a32b120febad5e0677c19d4cb532db73
Sep 08 15:25:44.822326 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.822302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drz7c\" (UniqueName: \"kubernetes.io/projected/bbdbbb1c-1d5e-4ede-8061-ca5347835316-kube-api-access-drz7c\") pod \"authorino-b54df94b4-9gvl5\" (UID: \"bbdbbb1c-1d5e-4ede-8061-ca5347835316\") " pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:44.831863 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.831816    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drz7c\" (UniqueName: \"kubernetes.io/projected/bbdbbb1c-1d5e-4ede-8061-ca5347835316-kube-api-access-drz7c\") pod \"authorino-b54df94b4-9gvl5\" (UID: \"bbdbbb1c-1d5e-4ede-8061-ca5347835316\") " pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:44.873246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:44.873214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:45.145790 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:45.145765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-9gvl5"]
Sep 08 15:25:45.148333 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:25:45.148309    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbdbbb1c_1d5e_4ede_8061_ca5347835316.slice/crio-ebb86bb105bd9c9d3d5c971295e412ae91c1448ecf1f826e24f40c8f4782bfc6 WatchSource:0}: Error finding container ebb86bb105bd9c9d3d5c971295e412ae91c1448ecf1f826e24f40c8f4782bfc6: Status 404 returned error can't find the container with id ebb86bb105bd9c9d3d5c971295e412ae91c1448ecf1f826e24f40c8f4782bfc6
Sep 08 15:25:45.452049 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:45.451980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268" event={"ID":"55cef3ff-3c94-4cac-af00-f8025af07dce","Type":"ContainerStarted","Data":"0f010278cf21cd1a930cee1044d59ba1a32b120febad5e0677c19d4cb532db73"}
Sep 08 15:25:45.453124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:45.453103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-9gvl5" event={"ID":"bbdbbb1c-1d5e-4ede-8061-ca5347835316","Type":"ContainerStarted","Data":"ebb86bb105bd9c9d3d5c971295e412ae91c1448ecf1f826e24f40c8f4782bfc6"}
Sep 08 15:25:46.458902 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:46.458871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268" event={"ID":"55cef3ff-3c94-4cac-af00-f8025af07dce","Type":"ContainerStarted","Data":"debf9e3a99d20819278618cfaa205ce211ff1179113f8a594134a04c1cd400e8"}
Sep 08 15:25:46.459226 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:46.459005    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:25:46.480665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:46.480621    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268" podStartSLOduration=1.139377263 podStartE2EDuration="2.480606534s" totalImagesPullingTime="1.341229271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:46.479090272 +0000 UTC m=+495.554382566" watchObservedRunningTime="2026-09-08 15:25:46.480606534 +0000 UTC m=+495.555898826"
Sep 08 15:25:49.150783 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:49.150748    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-9gvl5"]
Sep 08 15:25:49.473869 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:49.473816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-9gvl5" event={"ID":"bbdbbb1c-1d5e-4ede-8061-ca5347835316","Type":"ContainerStarted","Data":"f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601"}
Sep 08 15:25:49.491932 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:49.491886    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-9gvl5" podStartSLOduration=1.591597525 podStartE2EDuration="5.49186738s" totalImagesPullingTime="3.900269855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:49.490877609 +0000 UTC m=+498.566169902" watchObservedRunningTime="2026-09-08 15:25:49.49186738 +0000 UTC m=+498.567159678"
Sep 08 15:25:50.478602 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:50.478551    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-9gvl5" podUID="bbdbbb1c-1d5e-4ede-8061-ca5347835316" containerName="authorino" containerID="cri-o://f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601" gracePeriod=30
Sep 08 15:25:50.723504 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:50.723482    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:50.868417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:50.868353    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drz7c\" (UniqueName: \"kubernetes.io/projected/bbdbbb1c-1d5e-4ede-8061-ca5347835316-kube-api-access-drz7c\") pod \"bbdbbb1c-1d5e-4ede-8061-ca5347835316\" (UID: \"bbdbbb1c-1d5e-4ede-8061-ca5347835316\") "
Sep 08 15:25:50.870541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:50.870513    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdbbb1c-1d5e-4ede-8061-ca5347835316-kube-api-access-drz7c" (OuterVolumeSpecName: "kube-api-access-drz7c") pod "bbdbbb1c-1d5e-4ede-8061-ca5347835316" (UID: "bbdbbb1c-1d5e-4ede-8061-ca5347835316"). InnerVolumeSpecName "kube-api-access-drz7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:25:50.969495 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:50.969476    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drz7c\" (UniqueName: \"kubernetes.io/projected/bbdbbb1c-1d5e-4ede-8061-ca5347835316-kube-api-access-drz7c\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:25:51.483351 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.483313    2575 generic.go:356] "Generic (PLEG): container finished" podID="bbdbbb1c-1d5e-4ede-8061-ca5347835316" containerID="f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601" exitCode=0
Sep 08 15:25:51.483728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.483378    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-9gvl5" event={"ID":"bbdbbb1c-1d5e-4ede-8061-ca5347835316","Type":"ContainerDied","Data":"f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601"}
Sep 08 15:25:51.483728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.483413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-9gvl5" event={"ID":"bbdbbb1c-1d5e-4ede-8061-ca5347835316","Type":"ContainerDied","Data":"ebb86bb105bd9c9d3d5c971295e412ae91c1448ecf1f826e24f40c8f4782bfc6"}
Sep 08 15:25:51.483728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.483415    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-9gvl5"
Sep 08 15:25:51.483728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.483493    2575 scope.go:117] "RemoveContainer" containerID="f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601"
Sep 08 15:25:51.491906 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.491890    2575 scope.go:117] "RemoveContainer" containerID="f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601"
Sep 08 15:25:51.492179 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:25:51.492161    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601\": container with ID starting with f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601 not found: ID does not exist" containerID="f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601"
Sep 08 15:25:51.492223 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.492187    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601"} err="failed to get container status \"f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601\": rpc error: code = NotFound desc = could not find container \"f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601\": container with ID starting with f47551db78bc7f2f6aad92b0b33ef78740d1a46373eb5fdf988b4ec58e174601 not found: ID does not exist"
Sep 08 15:25:51.500665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.500644    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-9gvl5"]
Sep 08 15:25:51.504631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:51.504613    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-9gvl5"]
Sep 08 15:25:53.441459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:53.441418    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbdbbb1c-1d5e-4ede-8061-ca5347835316" path="/var/lib/kubelet/pods/bbdbbb1c-1d5e-4ede-8061-ca5347835316/volumes"
Sep 08 15:25:57.464286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:25:57.464259    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h5268"
Sep 08 15:26:16.620452 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.620419    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xlmb2"]
Sep 08 15:26:16.620918 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.620695    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbdbbb1c-1d5e-4ede-8061-ca5347835316" containerName="authorino"
Sep 08 15:26:16.620918 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.620705    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbdbbb1c-1d5e-4ede-8061-ca5347835316" containerName="authorino"
Sep 08 15:26:16.620918 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.620771    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbdbbb1c-1d5e-4ede-8061-ca5347835316" containerName="authorino"
Sep 08 15:26:16.625521 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.625506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:16.628531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.628511    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rvx67\""
Sep 08 15:26:16.632743 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.632725    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xlmb2"]
Sep 08 15:26:16.710975 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.710955    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xlmb2"]
Sep 08 15:26:16.711129 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:26:16.711113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nwpjh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2" podUID="ce3f2d2b-8af3-4a61-8603-34f5553917a5"
Sep 08 15:26:16.747864 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.747822    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpjh\" (UniqueName: \"kubernetes.io/projected/ce3f2d2b-8af3-4a61-8603-34f5553917a5-kube-api-access-nwpjh\") pod \"authorino-5dc7ccb5f-xlmb2\" (UID: \"ce3f2d2b-8af3-4a61-8603-34f5553917a5\") " pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:16.753734 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.753713    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-9qstc"]
Sep 08 15:26:16.756071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.756052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:16.759751 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.759735    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 15:26:16.766701 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.766678    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-9qstc"]
Sep 08 15:26:16.848450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.848424    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfrb\" (UniqueName: \"kubernetes.io/projected/a7c63665-0ecf-4745-a36b-a2f36bf9c789-kube-api-access-5lfrb\") pod \"authorino-687bd45585-9qstc\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") " pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:16.848541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.848458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpjh\" (UniqueName: \"kubernetes.io/projected/ce3f2d2b-8af3-4a61-8603-34f5553917a5-kube-api-access-nwpjh\") pod \"authorino-5dc7ccb5f-xlmb2\" (UID: \"ce3f2d2b-8af3-4a61-8603-34f5553917a5\") " pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:16.848541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.848485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7c63665-0ecf-4745-a36b-a2f36bf9c789-tls-cert\") pod \"authorino-687bd45585-9qstc\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") " pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:16.857292 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.857269    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-9qstc"]
Sep 08 15:26:16.857867 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:26:16.857816    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5lfrb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-9qstc" podUID="a7c63665-0ecf-4745-a36b-a2f36bf9c789"
Sep 08 15:26:16.863539 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.863520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpjh\" (UniqueName: \"kubernetes.io/projected/ce3f2d2b-8af3-4a61-8603-34f5553917a5-kube-api-access-nwpjh\") pod \"authorino-5dc7ccb5f-xlmb2\" (UID: \"ce3f2d2b-8af3-4a61-8603-34f5553917a5\") " pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:16.886745 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.886723    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bdf566758-vghgk"]
Sep 08 15:26:16.888836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.888822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:16.900097 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.900079    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bdf566758-vghgk"]
Sep 08 15:26:16.949301 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.949273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7c63665-0ecf-4745-a36b-a2f36bf9c789-tls-cert\") pod \"authorino-687bd45585-9qstc\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") " pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:16.949447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.949347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfrb\" (UniqueName: \"kubernetes.io/projected/a7c63665-0ecf-4745-a36b-a2f36bf9c789-kube-api-access-5lfrb\") pod \"authorino-687bd45585-9qstc\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") " pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:16.951781 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.951759    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7c63665-0ecf-4745-a36b-a2f36bf9c789-tls-cert\") pod \"authorino-687bd45585-9qstc\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") " pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:16.957519 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:16.957499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfrb\" (UniqueName: \"kubernetes.io/projected/a7c63665-0ecf-4745-a36b-a2f36bf9c789-kube-api-access-5lfrb\") pod \"authorino-687bd45585-9qstc\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") " pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:17.050617 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.050583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcw5\" (UniqueName: \"kubernetes.io/projected/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-kube-api-access-jmcw5\") pod \"authorino-7bdf566758-vghgk\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") " pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.050700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.050649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-tls-cert\") pod \"authorino-7bdf566758-vghgk\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") " pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.151892 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.151818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcw5\" (UniqueName: \"kubernetes.io/projected/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-kube-api-access-jmcw5\") pod \"authorino-7bdf566758-vghgk\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") " pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.151977 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.151900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-tls-cert\") pod \"authorino-7bdf566758-vghgk\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") " pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.154386 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.154362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-tls-cert\") pod \"authorino-7bdf566758-vghgk\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") " pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.162583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.162562    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcw5\" (UniqueName: \"kubernetes.io/projected/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-kube-api-access-jmcw5\") pod \"authorino-7bdf566758-vghgk\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") " pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.197413 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.197391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:26:17.469887 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.469863    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bdf566758-vghgk"]
Sep 08 15:26:17.472247 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:26:17.472214    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18a61aa8_e6fa_49c6_ae75_ce42490cf8a9.slice/crio-1c9e4aceb2060c18dde35f68a73a0acc60d8f09bf564f4cfb9d4adca2fa4982b WatchSource:0}: Error finding container 1c9e4aceb2060c18dde35f68a73a0acc60d8f09bf564f4cfb9d4adca2fa4982b: Status 404 returned error can't find the container with id 1c9e4aceb2060c18dde35f68a73a0acc60d8f09bf564f4cfb9d4adca2fa4982b
Sep 08 15:26:17.599419 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.599394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:17.599553 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.599390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdf566758-vghgk" event={"ID":"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9","Type":"ContainerStarted","Data":"1c9e4aceb2060c18dde35f68a73a0acc60d8f09bf564f4cfb9d4adca2fa4982b"}
Sep 08 15:26:17.599598 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.599581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:17.605581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.605563    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:17.608160 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.608145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:17.756477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.756429    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7c63665-0ecf-4745-a36b-a2f36bf9c789-tls-cert\") pod \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") "
Sep 08 15:26:17.756477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.756460    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lfrb\" (UniqueName: \"kubernetes.io/projected/a7c63665-0ecf-4745-a36b-a2f36bf9c789-kube-api-access-5lfrb\") pod \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\" (UID: \"a7c63665-0ecf-4745-a36b-a2f36bf9c789\") "
Sep 08 15:26:17.756793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.756509    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwpjh\" (UniqueName: \"kubernetes.io/projected/ce3f2d2b-8af3-4a61-8603-34f5553917a5-kube-api-access-nwpjh\") pod \"ce3f2d2b-8af3-4a61-8603-34f5553917a5\" (UID: \"ce3f2d2b-8af3-4a61-8603-34f5553917a5\") "
Sep 08 15:26:17.758529 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.758499    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c63665-0ecf-4745-a36b-a2f36bf9c789-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a7c63665-0ecf-4745-a36b-a2f36bf9c789" (UID: "a7c63665-0ecf-4745-a36b-a2f36bf9c789"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:17.758773 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.758754    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c63665-0ecf-4745-a36b-a2f36bf9c789-kube-api-access-5lfrb" (OuterVolumeSpecName: "kube-api-access-5lfrb") pod "a7c63665-0ecf-4745-a36b-a2f36bf9c789" (UID: "a7c63665-0ecf-4745-a36b-a2f36bf9c789"). InnerVolumeSpecName "kube-api-access-5lfrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:26:17.758823 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.758783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3f2d2b-8af3-4a61-8603-34f5553917a5-kube-api-access-nwpjh" (OuterVolumeSpecName: "kube-api-access-nwpjh") pod "ce3f2d2b-8af3-4a61-8603-34f5553917a5" (UID: "ce3f2d2b-8af3-4a61-8603-34f5553917a5"). InnerVolumeSpecName "kube-api-access-nwpjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:26:17.856986 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.856961    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwpjh\" (UniqueName: \"kubernetes.io/projected/ce3f2d2b-8af3-4a61-8603-34f5553917a5-kube-api-access-nwpjh\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:26:17.856986 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.856981    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7c63665-0ecf-4745-a36b-a2f36bf9c789-tls-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:26:17.857101 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:17.856990    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lfrb\" (UniqueName: \"kubernetes.io/projected/a7c63665-0ecf-4745-a36b-a2f36bf9c789-kube-api-access-5lfrb\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:26:18.605260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.605225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdf566758-vghgk" event={"ID":"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9","Type":"ContainerStarted","Data":"1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c"}
Sep 08 15:26:18.605260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.605252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9qstc"
Sep 08 15:26:18.605449 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.605405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xlmb2"
Sep 08 15:26:18.623213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.623171    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bdf566758-vghgk" podStartSLOduration=2.055492414 podStartE2EDuration="2.623157775s" totalImagesPullingTime="567.665361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:18.62172183 +0000 UTC m=+527.697014123" watchObservedRunningTime="2026-09-08 15:26:18.623157775 +0000 UTC m=+527.698450068"
Sep 08 15:26:18.651271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.651247    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xlmb2"]
Sep 08 15:26:18.658564 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.658541    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xlmb2"]
Sep 08 15:26:18.680551 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.680526    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-9qstc"]
Sep 08 15:26:18.685325 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:18.685301    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-9qstc"]
Sep 08 15:26:19.441047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:19.441019    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c63665-0ecf-4745-a36b-a2f36bf9c789" path="/var/lib/kubelet/pods/a7c63665-0ecf-4745-a36b-a2f36bf9c789/volumes"
Sep 08 15:26:19.441415 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:19.441211    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3f2d2b-8af3-4a61-8603-34f5553917a5" path="/var/lib/kubelet/pods/ce3f2d2b-8af3-4a61-8603-34f5553917a5/volumes"
Sep 08 15:26:59.953693 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:26:59.953656    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:27:07.455027 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:07.454987    2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6k5qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 15:27:07.455461 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:07.455063    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podUID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 15:27:09.832431 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:09.832401    2575 generic.go:356] "Generic (PLEG): container finished" podID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerID="1e76df51f995c1b187339eb49ac7bc90612ce6ad7ae607e6fa635422a6eadbab" exitCode=0
Sep 08 15:27:09.832930 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:09.832443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" event={"ID":"7170ec93-6665-4421-8ba3-9d6747ad0146","Type":"ContainerDied","Data":"1e76df51f995c1b187339eb49ac7bc90612ce6ad7ae607e6fa635422a6eadbab"}
Sep 08 15:27:09.832930 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:09.832793    2575 scope.go:117] "RemoveContainer" containerID="1e76df51f995c1b187339eb49ac7bc90612ce6ad7ae607e6fa635422a6eadbab"
Sep 08 15:27:09.833956 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:09.833902    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:27:09.834066 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:09.834014    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:27:09.834066 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:09.834059    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:27:10.838099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:10.838063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" event={"ID":"7170ec93-6665-4421-8ba3-9d6747ad0146","Type":"ContainerStarted","Data":"10b9681ddcfc656a6e7d5a4106ef9ae7778a84b6c1041cd5f3432b2e9e5bbc38"}
Sep 08 15:27:11.608669 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:11.608642    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:27:11.608872 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:11.608822    2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6k5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 08 15:27:11.608920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:11.608891    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podUID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 08 15:27:12.608939 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:12.608905    2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6k5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 08 15:27:12.609296 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:12.608966    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podUID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 08 15:27:13.608874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:13.608815    2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6k5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 08 15:27:13.609070 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:13.608901    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podUID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 08 15:27:14.609155 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:14.609117    2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6k5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 08 15:27:14.609536 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:14.609173    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podUID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 08 15:27:15.608814 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:15.608776    2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6k5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 08 15:27:15.609004 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:15.608836    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw" podUID="7170ec93-6665-4421-8ba3-9d6747ad0146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 08 15:27:16.612802 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:16.612769    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:27:16.613312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:16.613103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:27:16.613839 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:16.613818    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6k5qw"
Sep 08 15:27:18.008693 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:18.008653    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:27:25.486477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:25.486439    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:27:35.304096 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:35.304046    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:27:50.602452 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:50.602415    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:27:57.309079 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:27:57.308992    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:28:04.907423 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.907383    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"]
Sep 08 15:28:04.911085 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.911057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:04.913949 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.913920    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 15:28:04.915172 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.915148    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 15:28:04.915306 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.915152    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 15:28:04.915378 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.915222    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tmrxc\""
Sep 08 15:28:04.919925 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.919904    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"]
Sep 08 15:28:04.960145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.960120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rvx\" (UniqueName: \"kubernetes.io/projected/b271884c-a332-4576-ab7e-de755e4c7375-kube-api-access-69rvx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:04.960245 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.960166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:04.960245 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.960191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:04.960325 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.960300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:04.960379 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.960364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271884c-a332-4576-ab7e-de755e4c7375-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:04.960426 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:04.960412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.060917 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.060890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.060920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rvx\" (UniqueName: \"kubernetes.io/projected/b271884c-a332-4576-ab7e-de755e4c7375-kube-api-access-69rvx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.060945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.060966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.060987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061182 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.061026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271884c-a332-4576-ab7e-de755e4c7375-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061374 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.061351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.061387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.061493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.061447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.063443 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.063423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b271884c-a332-4576-ab7e-de755e4c7375-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.063614 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.063597    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271884c-a332-4576-ab7e-de755e4c7375-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.069261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.069234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rvx\" (UniqueName: \"kubernetes.io/projected/b271884c-a332-4576-ab7e-de755e4c7375-kube-api-access-69rvx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfc2\" (UID: \"b271884c-a332-4576-ab7e-de755e4c7375\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.223829 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.223771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:05.509234 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.509182    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"]
Sep 08 15:28:05.512013 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:28:05.511989    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb271884c_a332_4576_ab7e_de755e4c7375.slice/crio-ad38dd78198d2f0266e335df4461ffd4acd84f1b855a36b417a1a2b70c9c128a WatchSource:0}: Error finding container ad38dd78198d2f0266e335df4461ffd4acd84f1b855a36b417a1a2b70c9c128a: Status 404 returned error can't find the container with id ad38dd78198d2f0266e335df4461ffd4acd84f1b855a36b417a1a2b70c9c128a
Sep 08 15:28:05.513627 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:05.513607    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:28:06.079864 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:06.079804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2" event={"ID":"b271884c-a332-4576-ab7e-de755e4c7375","Type":"ContainerStarted","Data":"ad38dd78198d2f0266e335df4461ffd4acd84f1b855a36b417a1a2b70c9c128a"}
Sep 08 15:28:06.102429 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:06.102395    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:28:07.086018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:07.085973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2" event={"ID":"b271884c-a332-4576-ab7e-de755e4c7375","Type":"ContainerStarted","Data":"493c1aa7e6f5a622a4bc570820469229fb35649a2fe01ab64ffd9eadcde145b7"}
Sep 08 15:28:07.110377 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:07.110314    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2" podStartSLOduration=2.033711768 podStartE2EDuration="3.11029884s" totalImagesPullingTime="1.076587072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:07.109925635 +0000 UTC m=+636.185217931" watchObservedRunningTime="2026-09-08 15:28:07.11029884 +0000 UTC m=+636.185591133"
Sep 08 15:28:15.224704 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:15.224669    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:15.225194 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:15.224724    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:15.237663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:15.237642    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:16.140473 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:16.140438    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfc2"
Sep 08 15:28:17.604724 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.604693    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"]
Sep 08 15:28:17.607977 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.607954    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.611353 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.611331    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 15:28:17.622632 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.622611    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"]
Sep 08 15:28:17.657699 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.657679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.657798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.657715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.657798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.657735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.657934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.657829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a66a2f-a79c-4575-9d7d-18e82386cfdf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.657934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.657929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.658013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.657947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbbz\" (UniqueName: \"kubernetes.io/projected/84a66a2f-a79c-4575-9d7d-18e82386cfdf-kube-api-access-pwbbz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.758986 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.758953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a66a2f-a79c-4575-9d7d-18e82386cfdf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759103 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759103 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbbz\" (UniqueName: \"kubernetes.io/projected/84a66a2f-a79c-4575-9d7d-18e82386cfdf-kube-api-access-pwbbz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759103 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.759652 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.759517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.761315 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.761296    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a66a2f-a79c-4575-9d7d-18e82386cfdf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.761564 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.761544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a66a2f-a79c-4575-9d7d-18e82386cfdf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.769891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.769871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbbz\" (UniqueName: \"kubernetes.io/projected/84a66a2f-a79c-4575-9d7d-18e82386cfdf-kube-api-access-pwbbz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf\" (UID: \"84a66a2f-a79c-4575-9d7d-18e82386cfdf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:17.916757 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:17.916736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:18.188309 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:18.188282    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"]
Sep 08 15:28:18.191282 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:28:18.191258    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a66a2f_a79c_4575_9d7d_18e82386cfdf.slice/crio-05bd1a698df83c3aa71359f13b4d6f30267c2a00cbab2c0357b33416c51b67f3 WatchSource:0}: Error finding container 05bd1a698df83c3aa71359f13b4d6f30267c2a00cbab2c0357b33416c51b67f3: Status 404 returned error can't find the container with id 05bd1a698df83c3aa71359f13b4d6f30267c2a00cbab2c0357b33416c51b67f3
Sep 08 15:28:19.150916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:19.150874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf" event={"ID":"84a66a2f-a79c-4575-9d7d-18e82386cfdf","Type":"ContainerStarted","Data":"bb9b73bfb61b264fec486f10515804155075ecb77e338e0947ca1040817bf25e"}
Sep 08 15:28:19.151298 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:19.150926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf" event={"ID":"84a66a2f-a79c-4575-9d7d-18e82386cfdf","Type":"ContainerStarted","Data":"05bd1a698df83c3aa71359f13b4d6f30267c2a00cbab2c0357b33416c51b67f3"}
Sep 08 15:28:19.173810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:19.173753    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf" podStartSLOduration=1.92010703 podStartE2EDuration="2.173738025s" totalImagesPullingTime="253.630995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:19.172616837 +0000 UTC m=+648.247909131" watchObservedRunningTime="2026-09-08 15:28:19.173738025 +0000 UTC m=+648.249030318"
Sep 08 15:28:19.902463 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:19.902425    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:28:27.917014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:27.916979    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:27.917411 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:27.917033    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:27.929558 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:27.929532    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:28:28.203245 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:28:28.203167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf"
Sep 08 15:29:41.579023 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.578984    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bdf566758-vghgk"]
Sep 08 15:29:41.579618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.579161    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bdf566758-vghgk" podUID="18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" containerName="authorino" containerID="cri-o://1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c" gracePeriod=30
Sep 08 15:29:41.821855 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.821821    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:29:41.853921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.853866    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-tls-cert\") pod \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") "
Sep 08 15:29:41.853921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.853916    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmcw5\" (UniqueName: \"kubernetes.io/projected/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-kube-api-access-jmcw5\") pod \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\" (UID: \"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9\") "
Sep 08 15:29:41.856071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.856043    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-kube-api-access-jmcw5" (OuterVolumeSpecName: "kube-api-access-jmcw5") pod "18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" (UID: "18a61aa8-e6fa-49c6-ae75-ce42490cf8a9"). InnerVolumeSpecName "kube-api-access-jmcw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:29:41.863478 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.863455    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" (UID: "18a61aa8-e6fa-49c6-ae75-ce42490cf8a9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:29:41.955008 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.954987    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-tls-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.955092 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:41.955009    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcw5\" (UniqueName: \"kubernetes.io/projected/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9-kube-api-access-jmcw5\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:29:42.522640 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.522608    2575 generic.go:356] "Generic (PLEG): container finished" podID="18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" containerID="1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c" exitCode=0
Sep 08 15:29:42.522814 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.522656    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdf566758-vghgk"
Sep 08 15:29:42.522814 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.522687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdf566758-vghgk" event={"ID":"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9","Type":"ContainerDied","Data":"1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c"}
Sep 08 15:29:42.522814 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.522716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdf566758-vghgk" event={"ID":"18a61aa8-e6fa-49c6-ae75-ce42490cf8a9","Type":"ContainerDied","Data":"1c9e4aceb2060c18dde35f68a73a0acc60d8f09bf564f4cfb9d4adca2fa4982b"}
Sep 08 15:29:42.522814 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.522731    2575 scope.go:117] "RemoveContainer" containerID="1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c"
Sep 08 15:29:42.531203 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.531188    2575 scope.go:117] "RemoveContainer" containerID="1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c"
Sep 08 15:29:42.531468 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:29:42.531452    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c\": container with ID starting with 1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c not found: ID does not exist" containerID="1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c"
Sep 08 15:29:42.531505 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.531478    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c"} err="failed to get container status \"1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c\": rpc error: code = NotFound desc = could not find container \"1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c\": container with ID starting with 1fd3f95f86aca76646068134631da50bdb30a8205d298c739ff3309ce636103c not found: ID does not exist"
Sep 08 15:29:42.550827 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.550802    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bdf566758-vghgk"]
Sep 08 15:29:42.571610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:42.571582    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bdf566758-vghgk"]
Sep 08 15:29:43.441556 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:29:43.441521    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" path="/var/lib/kubelet/pods/18a61aa8-e6fa-49c6-ae75-ce42490cf8a9/volumes"
Sep 08 15:30:00.289088 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.289057    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"]
Sep 08 15:30:00.289434 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.289338    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" containerName="authorino"
Sep 08 15:30:00.289434 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.289348    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" containerName="authorino"
Sep 08 15:30:00.289434 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.289406    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a61aa8-e6fa-49c6-ae75-ce42490cf8a9" containerName="authorino"
Sep 08 15:30:00.292145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.292127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"
Sep 08 15:30:00.296418 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.296394    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9jgf\""
Sep 08 15:30:00.302205 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.302187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"
Sep 08 15:30:00.314031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.313998    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"]
Sep 08 15:30:00.600989 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:00.600889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"]
Sep 08 15:30:00.609359 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:00.609335    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6cb92e_3dd3_4801_b487_fed820d147e3.slice/crio-fc4c137195049e3483cb74598e5e73c5c77f88314710b3c241cb48c5580fccf8 WatchSource:0}: Error finding container fc4c137195049e3483cb74598e5e73c5c77f88314710b3c241cb48c5580fccf8: Status 404 returned error can't find the container with id fc4c137195049e3483cb74598e5e73c5c77f88314710b3c241cb48c5580fccf8
Sep 08 15:30:01.249830 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.249797    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"]
Sep 08 15:30:01.254302 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.254279    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.257895 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.257878    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-34fed8-openshift-default-dockercfg-g87cx\""
Sep 08 15:30:01.282004 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.281984    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"]
Sep 08 15:30:01.397680 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-envoy\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-certs\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2116d54-3f72-4a7e-960d-7582e423136b-istio-podinfo\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trhxk\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-kube-api-access-trhxk\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397813    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2116d54-3f72-4a7e-960d-7582e423136b-istiod-ca-cert\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-socket\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-credential-socket\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-data\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.398048 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.397939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-istio-token\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499127 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2116d54-3f72-4a7e-960d-7582e423136b-istiod-ca-cert\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-socket\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-credential-socket\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-data\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-istio-token\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-envoy\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499318 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-certs\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499635 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2116d54-3f72-4a7e-960d-7582e423136b-istio-podinfo\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499635 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trhxk\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-kube-api-access-trhxk\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499635 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499561    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-socket\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499658    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-data\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-certs\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499897 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-credential-socket\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.499939 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.499904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2116d54-3f72-4a7e-960d-7582e423136b-istiod-ca-cert\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.501901 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.501879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-envoy\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.502073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.502056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2116d54-3f72-4a7e-960d-7582e423136b-istio-podinfo\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.533269 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.533249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trhxk\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-kube-api-access-trhxk\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.539925 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.539877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-istio-token\") pod \"e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") " pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.564356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.564338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:01.614334 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.614213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68" event={"ID":"0c6cb92e-3dd3-4801-b487-fed820d147e3","Type":"ContainerStarted","Data":"fc4c137195049e3483cb74598e5e73c5c77f88314710b3c241cb48c5580fccf8"}
Sep 08 15:30:01.862734 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.862670    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"]
Sep 08 15:30:01.865783 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:01.865761    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2116d54_3f72_4a7e_960d_7582e423136b.slice/crio-f87be1a13b72305575375decb3b2d6e3c2676af47419b5866dafe86c49fda12f WatchSource:0}: Error finding container f87be1a13b72305575375decb3b2d6e3c2676af47419b5866dafe86c49fda12f: Status 404 returned error can't find the container with id f87be1a13b72305575375decb3b2d6e3c2676af47419b5866dafe86c49fda12f
Sep 08 15:30:01.867816 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.867777    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:01.867916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.867865    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:01.867916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:01.867897    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:02.620104 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.620072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" event={"ID":"c2116d54-3f72-4a7e-960d-7582e423136b","Type":"ContainerStarted","Data":"01d1cfbf11d9604ae7b5b8c3e6bccd0eb0f63fd854681a2a1a95c48a378e063c"}
Sep 08 15:30:02.620464 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.620120    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" event={"ID":"c2116d54-3f72-4a7e-960d-7582e423136b","Type":"ContainerStarted","Data":"f87be1a13b72305575375decb3b2d6e3c2676af47419b5866dafe86c49fda12f"}
Sep 08 15:30:02.681390 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.681345    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podStartSLOduration=1.681330436 podStartE2EDuration="1.681330436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:02.677902829 +0000 UTC m=+751.753195139" watchObservedRunningTime="2026-09-08 15:30:02.681330436 +0000 UTC m=+751.756622720"
Sep 08 15:30:02.780446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.780421    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-daf787"]
Sep 08 15:30:02.783743 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.783727    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:02.788536 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.788518    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-md8bz\""
Sep 08 15:30:02.797679 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.797660    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-daf787"]
Sep 08 15:30:02.911114 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:02.911089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrb7w\" (UniqueName: \"kubernetes.io/projected/7cc6acec-ae18-42a6-993b-27b1d53dbaf5-kube-api-access-wrb7w\") pod \"test-curl-7761-daf787\" (UID: \"7cc6acec-ae18-42a6-993b-27b1d53dbaf5\") " pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:03.012419 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.012390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrb7w\" (UniqueName: \"kubernetes.io/projected/7cc6acec-ae18-42a6-993b-27b1d53dbaf5-kube-api-access-wrb7w\") pod \"test-curl-7761-daf787\" (UID: \"7cc6acec-ae18-42a6-993b-27b1d53dbaf5\") " pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:03.050090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.050061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrb7w\" (UniqueName: \"kubernetes.io/projected/7cc6acec-ae18-42a6-993b-27b1d53dbaf5-kube-api-access-wrb7w\") pod \"test-curl-7761-daf787\" (UID: \"7cc6acec-ae18-42a6-993b-27b1d53dbaf5\") " pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:03.092936 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.092907    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:03.380217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.380105    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-daf787"]
Sep 08 15:30:03.385613 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:03.385585    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cc6acec_ae18_42a6_993b_27b1d53dbaf5.slice/crio-d14b2cf7aa28d1db09d91b064755c4fb39d22f4055af28406297d4f483ee6382 WatchSource:0}: Error finding container d14b2cf7aa28d1db09d91b064755c4fb39d22f4055af28406297d4f483ee6382: Status 404 returned error can't find the container with id d14b2cf7aa28d1db09d91b064755c4fb39d22f4055af28406297d4f483ee6382
Sep 08 15:30:03.565387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.565358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:03.571346 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.571313    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:03.626287 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.626250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-daf787" event={"ID":"7cc6acec-ae18-42a6-993b-27b1d53dbaf5","Type":"ContainerStarted","Data":"d14b2cf7aa28d1db09d91b064755c4fb39d22f4055af28406297d4f483ee6382"}
Sep 08 15:30:03.626764 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.626704    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:03.627457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:03.627435    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:04.941659 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:04.941628    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:30:05.645650 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:05.645594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68" event={"ID":"0c6cb92e-3dd3-4801-b487-fed820d147e3","Type":"ContainerStarted","Data":"7e90606809b51929e5417d14c67f46d88f37135a9e027f8d32b0dc71af2a40b4"}
Sep 08 15:30:05.714268 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:05.714209    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68" podStartSLOduration=1.826662048 podStartE2EDuration="5.714191394s" totalImagesPullingTime="3.887529346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:05.7126735 +0000 UTC m=+754.787965813" watchObservedRunningTime="2026-09-08 15:30:05.714191394 +0000 UTC m=+754.789483687"
Sep 08 15:30:06.651743 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:06.651705    2575 generic.go:356] "Generic (PLEG): container finished" podID="0c6cb92e-3dd3-4801-b487-fed820d147e3" containerID="7e90606809b51929e5417d14c67f46d88f37135a9e027f8d32b0dc71af2a40b4" exitCode=0
Sep 08 15:30:06.652153 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:06.651789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68" event={"ID":"0c6cb92e-3dd3-4801-b487-fed820d147e3","Type":"ContainerDied","Data":"7e90606809b51929e5417d14c67f46d88f37135a9e027f8d32b0dc71af2a40b4"}
Sep 08 15:30:06.653478 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:06.653449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-daf787" event={"ID":"7cc6acec-ae18-42a6-993b-27b1d53dbaf5","Type":"ContainerStarted","Data":"5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d"}
Sep 08 15:30:07.782425 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:07.782402    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"
Sep 08 15:30:07.998253 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:07.998161    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-daf787" podStartSLOduration=3.563440253 podStartE2EDuration="5.998146279s" totalImagesPullingTime="2.434706026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:06.744929376 +0000 UTC m=+755.820221669" watchObservedRunningTime="2026-09-08 15:30:07.998146279 +0000 UTC m=+757.073438573"
Sep 08 15:30:08.663749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:08.663671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68" event={"ID":"0c6cb92e-3dd3-4801-b487-fed820d147e3","Type":"ContainerDied","Data":"fc4c137195049e3483cb74598e5e73c5c77f88314710b3c241cb48c5580fccf8"}
Sep 08 15:30:08.663749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:08.663704    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-rqt68"
Sep 08 15:30:08.663749 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:08.663724    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4c137195049e3483cb74598e5e73c5c77f88314710b3c241cb48c5580fccf8"
Sep 08 15:30:09.076069 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.076000    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-daf787"]
Sep 08 15:30:09.076456 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.076160    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-daf787" podUID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" containerName="test-curl-7761-daf787" containerID="cri-o://5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d" gracePeriod=2
Sep 08 15:30:09.128175 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.128148    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-daf787"]
Sep 08 15:30:09.311535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.311504    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-234442"]
Sep 08 15:30:09.311894 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.311877    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6cb92e-3dd3-4801-b487-fed820d147e3" containerName="cleanup"
Sep 08 15:30:09.311985 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.311895    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6cb92e-3dd3-4801-b487-fed820d147e3" containerName="cleanup"
Sep 08 15:30:09.311985 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.311927    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" containerName="test-curl-7761-daf787"
Sep 08 15:30:09.311985 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.311936    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" containerName="test-curl-7761-daf787"
Sep 08 15:30:09.312131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.312017    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6cb92e-3dd3-4801-b487-fed820d147e3" containerName="cleanup"
Sep 08 15:30:09.312131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.312029    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" containerName="test-curl-7761-daf787"
Sep 08 15:30:09.315129 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.315109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:09.320605 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:09.320580    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-234442\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:09.323999 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:09.323977    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-daf787\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:09.367642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.367585    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-234442"]
Sep 08 15:30:09.469837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.469810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrlf\" (UniqueName: \"kubernetes.io/projected/f380f6dc-534c-4856-b885-3b9965635dd2-kube-api-access-8qrlf\") pod \"test-curl-7761-234442\" (UID: \"f380f6dc-534c-4856-b885-3b9965635dd2\") " pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:09.570420 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.570383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrlf\" (UniqueName: \"kubernetes.io/projected/f380f6dc-534c-4856-b885-3b9965635dd2-kube-api-access-8qrlf\") pod \"test-curl-7761-234442\" (UID: \"f380f6dc-534c-4856-b885-3b9965635dd2\") " pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:09.603030 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.603003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrlf\" (UniqueName: \"kubernetes.io/projected/f380f6dc-534c-4856-b885-3b9965635dd2-kube-api-access-8qrlf\") pod \"test-curl-7761-234442\" (UID: \"f380f6dc-534c-4856-b885-3b9965635dd2\") " pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:09.623762 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.623711    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:09.928591 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:09.928567    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-234442"]
Sep 08 15:30:09.929004 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:09.928982    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf380f6dc_534c_4856_b885_3b9965635dd2.slice/crio-d56f1b1e91b2b7cb16ec4a8a2e3875382199347223d70a4a86a7a8f95f4aa4f0 WatchSource:0}: Error finding container d56f1b1e91b2b7cb16ec4a8a2e3875382199347223d70a4a86a7a8f95f4aa4f0: Status 404 returned error can't find the container with id d56f1b1e91b2b7cb16ec4a8a2e3875382199347223d70a4a86a7a8f95f4aa4f0
Sep 08 15:30:10.675186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:10.675146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-234442" event={"ID":"f380f6dc-534c-4856-b885-3b9965635dd2","Type":"ContainerStarted","Data":"575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04"}
Sep 08 15:30:10.675822 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:10.675201    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-234442" event={"ID":"f380f6dc-534c-4856-b885-3b9965635dd2","Type":"ContainerStarted","Data":"d56f1b1e91b2b7cb16ec4a8a2e3875382199347223d70a4a86a7a8f95f4aa4f0"}
Sep 08 15:30:10.718352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:10.718305    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-234442" podStartSLOduration=1.718288302 podStartE2EDuration="1.718288302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:10.715312747 +0000 UTC m=+759.790605056" watchObservedRunningTime="2026-09-08 15:30:10.718288302 +0000 UTC m=+759.793580596"
Sep 08 15:30:11.261153 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.261130    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:11.383924 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.383900    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrb7w\" (UniqueName: \"kubernetes.io/projected/7cc6acec-ae18-42a6-993b-27b1d53dbaf5-kube-api-access-wrb7w\") pod \"7cc6acec-ae18-42a6-993b-27b1d53dbaf5\" (UID: \"7cc6acec-ae18-42a6-993b-27b1d53dbaf5\") "
Sep 08 15:30:11.386217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.386184    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc6acec-ae18-42a6-993b-27b1d53dbaf5-kube-api-access-wrb7w" (OuterVolumeSpecName: "kube-api-access-wrb7w") pod "7cc6acec-ae18-42a6-993b-27b1d53dbaf5" (UID: "7cc6acec-ae18-42a6-993b-27b1d53dbaf5"). InnerVolumeSpecName "kube-api-access-wrb7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:30:11.441354 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.441324    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" path="/var/lib/kubelet/pods/7cc6acec-ae18-42a6-993b-27b1d53dbaf5/volumes"
Sep 08 15:30:11.485158 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.485136    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrb7w\" (UniqueName: \"kubernetes.io/projected/7cc6acec-ae18-42a6-993b-27b1d53dbaf5-kube-api-access-wrb7w\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:30:11.675157 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.675126    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-234442"]
Sep 08 15:30:11.680010 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.679979    2575 generic.go:356] "Generic (PLEG): container finished" podID="7cc6acec-ae18-42a6-993b-27b1d53dbaf5" containerID="5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d" exitCode=137
Sep 08 15:30:11.680380 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.680024    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-daf787"
Sep 08 15:30:11.680380 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.680076    2575 scope.go:117] "RemoveContainer" containerID="5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d"
Sep 08 15:30:11.682350 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.682329    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-234442"]
Sep 08 15:30:11.690345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.690326    2575 scope.go:117] "RemoveContainer" containerID="5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d"
Sep 08 15:30:11.690719 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:11.690699    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d\": container with ID starting with 5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d not found: ID does not exist" containerID="5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d"
Sep 08 15:30:11.690785 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:11.690729    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d"} err="failed to get container status \"5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d\": rpc error: code = NotFound desc = could not find container \"5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d\": container with ID starting with 5f0d66568d10a4fb059bbec02d7ac4f9a15dcc24d866eabe56227afd50013c1d not found: ID does not exist"
Sep 08 15:30:12.686077 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.686041    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-234442" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" containerName="test-curl-7761-234442" containerID="cri-o://575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04" gracePeriod=2
Sep 08 15:30:12.694134 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:12.694096    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-234442\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:12.713287 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.713258    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"]
Sep 08 15:30:12.713598 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.713586    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" containerName="test-curl-7761-234442"
Sep 08 15:30:12.713642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.713601    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" containerName="test-curl-7761-234442"
Sep 08 15:30:12.713678 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.713667    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" containerName="test-curl-7761-234442"
Sep 08 15:30:12.718082 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.718064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.722297 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.722281    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8debe9-openshift-default-dockercfg-sc7sg\""
Sep 08 15:30:12.787664 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.787640    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"]
Sep 08 15:30:12.795299 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-envoy\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-socket\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rstl\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-kube-api-access-2rstl\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795564 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-certs\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795564 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c57828-a594-46d4-8b8e-91f6da343297-istio-podinfo\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795564 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c57828-a594-46d4-8b8e-91f6da343297-istiod-ca-cert\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795742 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-credential-socket\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795742 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-data\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.795742 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.795735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-istio-token\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.841041 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:12.840991    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-234442\" is forbidden: User \"system:node:ip-10-0-142-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-52.ec2.internal' and this object" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:12.896908 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.896884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-credential-socket\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.896912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-data\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.896930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-istio-token\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.896973    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-envoy\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.896992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-socket\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rstl\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-kube-api-access-2rstl\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897252 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-certs\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897252 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c57828-a594-46d4-8b8e-91f6da343297-istio-podinfo\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897252 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c57828-a594-46d4-8b8e-91f6da343297-istiod-ca-cert\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897398 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-credential-socket\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897398 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897355    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-data\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897556 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897525    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-certs\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897650    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-socket\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.897925 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.897810    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c57828-a594-46d4-8b8e-91f6da343297-istiod-ca-cert\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.899350 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.899331    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-envoy\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.899758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.899743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c57828-a594-46d4-8b8e-91f6da343297-istio-podinfo\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.916004 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.915977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rstl\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-kube-api-access-2rstl\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:12.917174 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:12.917158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-istio-token\") pod \"e2e-autores-8debe9-openshift-default-796d556b76-6jlnz\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") " pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:13.027376 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.027324    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:13.311388 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.311365    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"]
Sep 08 15:30:13.313248 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:13.313217    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c57828_a594_46d4_8b8e_91f6da343297.slice/crio-3fa825a6f16c21df8f23df7f59e2025d066f91c75e407b10a031d15ed93f3303 WatchSource:0}: Error finding container 3fa825a6f16c21df8f23df7f59e2025d066f91c75e407b10a031d15ed93f3303: Status 404 returned error can't find the container with id 3fa825a6f16c21df8f23df7f59e2025d066f91c75e407b10a031d15ed93f3303
Sep 08 15:30:13.315423 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.315388    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:13.315526 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.315466    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:13.315526 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.315511    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:13.692648 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.692609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" event={"ID":"31c57828-a594-46d4-8b8e-91f6da343297","Type":"ContainerStarted","Data":"837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"}
Sep 08 15:30:13.693047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.692672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" event={"ID":"31c57828-a594-46d4-8b8e-91f6da343297","Type":"ContainerStarted","Data":"3fa825a6f16c21df8f23df7f59e2025d066f91c75e407b10a031d15ed93f3303"}
Sep 08 15:30:13.731943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:13.731898    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podStartSLOduration=1.731883191 podStartE2EDuration="1.731883191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:13.728923551 +0000 UTC m=+762.804215863" watchObservedRunningTime="2026-09-08 15:30:13.731883191 +0000 UTC m=+762.807175483"
Sep 08 15:30:14.027990 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:14.027901    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:14.033420 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:14.033397    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:14.699382 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:14.699346    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:14.705578 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:14.705543    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:14.917880 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:14.917859    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:15.017793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.017712    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qrlf\" (UniqueName: \"kubernetes.io/projected/f380f6dc-534c-4856-b885-3b9965635dd2-kube-api-access-8qrlf\") pod \"f380f6dc-534c-4856-b885-3b9965635dd2\" (UID: \"f380f6dc-534c-4856-b885-3b9965635dd2\") "
Sep 08 15:30:15.020050 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.020020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f380f6dc-534c-4856-b885-3b9965635dd2-kube-api-access-8qrlf" (OuterVolumeSpecName: "kube-api-access-8qrlf") pod "f380f6dc-534c-4856-b885-3b9965635dd2" (UID: "f380f6dc-534c-4856-b885-3b9965635dd2"). InnerVolumeSpecName "kube-api-access-8qrlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:30:15.118984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.118950    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrlf\" (UniqueName: \"kubernetes.io/projected/f380f6dc-534c-4856-b885-3b9965635dd2-kube-api-access-8qrlf\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:30:15.441347 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.441314    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f380f6dc-534c-4856-b885-3b9965635dd2" path="/var/lib/kubelet/pods/f380f6dc-534c-4856-b885-3b9965635dd2/volumes"
Sep 08 15:30:15.704757 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.704674    2575 generic.go:356] "Generic (PLEG): container finished" podID="f380f6dc-534c-4856-b885-3b9965635dd2" containerID="575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04" exitCode=137
Sep 08 15:30:15.704757 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.704715    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-234442"
Sep 08 15:30:15.705250 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.704789    2575 scope.go:117] "RemoveContainer" containerID="575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04"
Sep 08 15:30:15.713145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.713131    2575 scope.go:117] "RemoveContainer" containerID="575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04"
Sep 08 15:30:15.713393 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:15.713374    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04\": container with ID starting with 575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04 not found: ID does not exist" containerID="575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04"
Sep 08 15:30:15.713453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:15.713401    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04"} err="failed to get container status \"575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04\": rpc error: code = NotFound desc = could not find container \"575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04\": container with ID starting with 575e33cf2e0e0ccb60effbb66ea7e881e42c99642dd1e6e2053517a0a4350e04 not found: ID does not exist"
Sep 08 15:30:21.700862 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:21.700817    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:30:22.313778 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.313734    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"]
Sep 08 15:30:22.321771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.321740    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.325404 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.325381    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2xlj4\""
Sep 08 15:30:22.326321 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.326301    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-425c6986-serving-cert\""
Sep 08 15:30:22.327251 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.327148    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:30:22.327397 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.327366    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:30:22.327733 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.327708    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:30:22.342434 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.342412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"]
Sep 08 15:30:22.485648 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.485609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.485864 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.485668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5b175d-117b-43ef-a35b-12a36d0923df-openshift-service-ca\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.485864 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.485720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsv4\" (UniqueName: \"kubernetes.io/projected/ac5b175d-117b-43ef-a35b-12a36d0923df-kube-api-access-jpsv4\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.485864 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.485760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-metrics-certs\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.587149 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.587068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5b175d-117b-43ef-a35b-12a36d0923df-openshift-service-ca\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.587149 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.587127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsv4\" (UniqueName: \"kubernetes.io/projected/ac5b175d-117b-43ef-a35b-12a36d0923df-kube-api-access-jpsv4\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.587364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.587178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-metrics-certs\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.587364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.587279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.587486 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:22.587389    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-serving-cert: secret "maas-api-e2e-ait-425c6986-serving-cert" not found
Sep 08 15:30:22.587486 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:22.587450    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls podName:ac5b175d-117b-43ef-a35b-12a36d0923df nodeName:}" failed. No retries permitted until 2026-09-08 15:30:23.087429325 +0000 UTC m=+772.162721624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls") pod "maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" (UID: "ac5b175d-117b-43ef-a35b-12a36d0923df") : secret "maas-api-e2e-ait-425c6986-serving-cert" not found
Sep 08 15:30:22.587886 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.587860    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5b175d-117b-43ef-a35b-12a36d0923df-openshift-service-ca\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.589971 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.589947    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-metrics-certs\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:22.603278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:22.603254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsv4\" (UniqueName: \"kubernetes.io/projected/ac5b175d-117b-43ef-a35b-12a36d0923df-kube-api-access-jpsv4\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:23.092803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.092290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:23.097197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.095952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls\") pod \"maas-api-e2e-ait-425c6986-686cfd77d-4bbvt\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:23.236590 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.236558    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:23.583867 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.583819    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"]
Sep 08 15:30:23.747195 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.747155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" event={"ID":"ac5b175d-117b-43ef-a35b-12a36d0923df","Type":"ContainerStarted","Data":"a96c3b4954f494b0ee10102ea6e14e2e55ddd61a61d7936b6edf7a57b30985d2"}
Sep 08 15:30:23.748825 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.748799    2575 generic.go:356] "Generic (PLEG): container finished" podID="c2116d54-3f72-4a7e-960d-7582e423136b" containerID="01d1cfbf11d9604ae7b5b8c3e6bccd0eb0f63fd854681a2a1a95c48a378e063c" exitCode=0
Sep 08 15:30:23.748974 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.748871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" event={"ID":"c2116d54-3f72-4a7e-960d-7582e423136b","Type":"ContainerDied","Data":"01d1cfbf11d9604ae7b5b8c3e6bccd0eb0f63fd854681a2a1a95c48a378e063c"}
Sep 08 15:30:23.749206 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.749193    2575 scope.go:117] "RemoveContainer" containerID="01d1cfbf11d9604ae7b5b8c3e6bccd0eb0f63fd854681a2a1a95c48a378e063c"
Sep 08 15:30:23.750131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.750094    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:23.750238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.750173    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:23.750238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:23.750205    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:24.264516 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.264485    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"]
Sep 08 15:30:24.266918 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.266902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.277873 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.274092    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-425c6986\""
Sep 08 15:30:24.294871 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.288826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"]
Sep 08 15:30:24.416415 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.416348    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl9k5\" (UniqueName: \"kubernetes.io/projected/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-kube-api-access-jl9k5\") pod \"payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.416639 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.416514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.517741 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.517636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9k5\" (UniqueName: \"kubernetes.io/projected/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-kube-api-access-jl9k5\") pod \"payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.518121 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.518097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.519159 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.519137    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.555352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.555331    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9k5\" (UniqueName: \"kubernetes.io/projected/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-kube-api-access-jl9k5\") pod \"payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.587537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.587503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:24.756024 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.755991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" event={"ID":"c2116d54-3f72-4a7e-960d-7582e423136b","Type":"ContainerStarted","Data":"ae3db3c218b985b0ffa37d8e27084fdddbfcf81e1867188cf7ec69abdd656400"}
Sep 08 15:30:24.963314 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:24.963288    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"]
Sep 08 15:30:24.965319 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:24.965292    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a4d313_44b3_4dfc_afd4_9a52f3e602c3.slice/crio-4c1343b6031da3448f49969d897e382ab887e8a5d6e5b5c0b364d49b75fad743 WatchSource:0}: Error finding container 4c1343b6031da3448f49969d897e382ab887e8a5d6e5b5c0b364d49b75fad743: Status 404 returned error can't find the container with id 4c1343b6031da3448f49969d897e382ab887e8a5d6e5b5c0b364d49b75fad743
Sep 08 15:30:25.564721 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:25.564625    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:25.565215 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:25.565119    2575 patch_prober.go:28] interesting pod/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 08 15:30:25.565286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:25.565262    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 08 15:30:25.765383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:25.765345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp" event={"ID":"38a4d313-44b3-4dfc-afd4-9a52f3e602c3","Type":"ContainerStarted","Data":"4c1343b6031da3448f49969d897e382ab887e8a5d6e5b5c0b364d49b75fad743"}
Sep 08 15:30:26.565383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:26.565343    2575 patch_prober.go:28] interesting pod/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 08 15:30:26.565827 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:26.565406    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 08 15:30:27.565470 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:27.565433    2575 patch_prober.go:28] interesting pod/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 08 15:30:27.565941 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:27.565493    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 08 15:30:28.565264 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:28.565218    2575 patch_prober.go:28] interesting pod/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 08 15:30:28.565408 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:28.565291    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 08 15:30:29.565041 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:29.565000    2575 patch_prober.go:28] interesting pod/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 08 15:30:29.565505 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:29.565077    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 08 15:30:30.568149 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.568109    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:30.568534 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.568382    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:30.569235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.569216    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:30:30.741010 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.740979    2575 patch_prober.go:28] interesting pod/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 15:30:30.741161 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.741032    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 15:30:30.823983 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.823941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" event={"ID":"ac5b175d-117b-43ef-a35b-12a36d0923df","Type":"ContainerStarted","Data":"300babfd8dd10c34adece1a08e69744d690f08fdb8751ccc410648885d98a9cf"}
Sep 08 15:30:30.824160 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.824074    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:30.825582 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.825547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp" event={"ID":"38a4d313-44b3-4dfc-afd4-9a52f3e602c3","Type":"ContainerStarted","Data":"b68ad0a6a2580641b474f18155b04c0fc262db4e027217d646066088e53f02b7"}
Sep 08 15:30:30.862165 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.862119    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" podStartSLOduration=2.103195937 podStartE2EDuration="8.862106297s" totalImagesPullingTime="6.75891036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:30.861987851 +0000 UTC m=+779.937280164" watchObservedRunningTime="2026-09-08 15:30:30.862106297 +0000 UTC m=+779.937398589"
Sep 08 15:30:30.927894 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:30.927837    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp" podStartSLOduration=1.538725476 podStartE2EDuration="6.927825697s" totalImagesPullingTime="5.389100221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:30.926613912 +0000 UTC m=+780.001906217" watchObservedRunningTime="2026-09-08 15:30:30.927825697 +0000 UTC m=+780.003118052"
Sep 08 15:30:31.830250 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:31.830173    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:32.836326 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:32.836298    2575 generic.go:356] "Generic (PLEG): container finished" podID="31c57828-a594-46d4-8b8e-91f6da343297" containerID="837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c" exitCode=0
Sep 08 15:30:32.836837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:32.836369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" event={"ID":"31c57828-a594-46d4-8b8e-91f6da343297","Type":"ContainerDied","Data":"837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"}
Sep 08 15:30:32.836837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:32.836784    2575 scope.go:117] "RemoveContainer" containerID="837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"
Sep 08 15:30:32.837771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:32.837745    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:32.837832 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:32.837818    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:32.837893 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:32.837874    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:30:33.701680 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.701638    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"]
Sep 08 15:30:33.704145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.704124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.709111 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.709091    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-d920a3f5-kserve-self-signed-certs\""
Sep 08 15:30:33.720325 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.720304    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"]
Sep 08 15:30:33.805969 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.805927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-model-cache\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.805969 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.805974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-home\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.806218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.805999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d687e584-155e-409c-ab9d-97a446791cd7-tls-certs\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.806218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.806030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxj8\" (UniqueName: \"kubernetes.io/projected/d687e584-155e-409c-ab9d-97a446791cd7-kube-api-access-xsxj8\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.806218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.806061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-tmp-dir\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.806218 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.806099    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-dshm\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.843203 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.843166    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" event={"ID":"31c57828-a594-46d4-8b8e-91f6da343297","Type":"ContainerStarted","Data":"337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c"}
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.908341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-home\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.908388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d687e584-155e-409c-ab9d-97a446791cd7-tls-certs\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.908416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxj8\" (UniqueName: \"kubernetes.io/projected/d687e584-155e-409c-ab9d-97a446791cd7-kube-api-access-xsxj8\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.908452    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-tmp-dir\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.908496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-dshm\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.908631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-model-cache\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.909108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-model-cache\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.909201    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-home\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.909966 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.909503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-tmp-dir\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.915487 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.915460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-dshm\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.918608 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.918572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d687e584-155e-409c-ab9d-97a446791cd7-tls-certs\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:33.929833 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:33.929814    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxj8\" (UniqueName: \"kubernetes.io/projected/d687e584-155e-409c-ab9d-97a446791cd7-kube-api-access-xsxj8\") pod \"e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") " pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:34.013608 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.013511    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:34.027780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.027685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:34.027934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.027873    2575 patch_prober.go:28] interesting pod/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 08 15:30:34.027999 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.027924    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 08 15:30:34.337265 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:34.337236    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd687e584_155e_409c_ab9d_97a446791cd7.slice/crio-114cabfddc77a278d061052d6bfef28f37541842555bbc4a9e8efb5246586868 WatchSource:0}: Error finding container 114cabfddc77a278d061052d6bfef28f37541842555bbc4a9e8efb5246586868: Status 404 returned error can't find the container with id 114cabfddc77a278d061052d6bfef28f37541842555bbc4a9e8efb5246586868
Sep 08 15:30:34.340375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.340353    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"]
Sep 08 15:30:34.849787 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.849750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7" event={"ID":"d687e584-155e-409c-ab9d-97a446791cd7","Type":"ContainerStarted","Data":"2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b"}
Sep 08 15:30:34.849787 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:34.849793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7" event={"ID":"d687e584-155e-409c-ab9d-97a446791cd7","Type":"ContainerStarted","Data":"114cabfddc77a278d061052d6bfef28f37541842555bbc4a9e8efb5246586868"}
Sep 08 15:30:35.028322 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:35.028283    2575 patch_prober.go:28] interesting pod/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 08 15:30:35.028489 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:35.028359    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 08 15:30:36.027817 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:36.027778    2575 patch_prober.go:28] interesting pod/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 08 15:30:36.028222 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:36.027863    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 08 15:30:36.835170 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:36.835144    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:30:36.912962 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:36.912917    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7" podStartSLOduration=3.9129025410000002 podStartE2EDuration="3.912902541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:34.912018706 +0000 UTC m=+783.987311004" watchObservedRunningTime="2026-09-08 15:30:36.912902541 +0000 UTC m=+785.988194837"
Sep 08 15:30:37.028363 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:37.028329    2575 patch_prober.go:28] interesting pod/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 08 15:30:37.028732 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:37.028378    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 08 15:30:38.028396 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:38.028359    2575 patch_prober.go:28] interesting pod/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 08 15:30:38.028809 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:38.028413    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 08 15:30:38.060485 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:38.060450    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:30:39.031240 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:39.031212    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:39.031633 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:39.031615    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:39.032385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:39.032369    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:30:42.838557 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:42.838521    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:30:44.014303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:44.014277    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:44.014303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:44.014314    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:44.026974 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:44.026948    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:44.909837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:44.909807    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:30:46.920583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:46.920549    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"]
Sep 08 15:30:46.922979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:46.922964    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:46.926029 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:46.926009    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e39762-serving-cert\""
Sep 08 15:30:46.949914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:46.949886    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"]
Sep 08 15:30:47.024054 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.024025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5s9\" (UniqueName: \"kubernetes.io/projected/2242137b-d234-4c2f-ad8a-eb48e972dcef-kube-api-access-kc5s9\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.024181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.024079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.024181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.024109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2242137b-d234-4c2f-ad8a-eb48e972dcef-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.024181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.024129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-metrics-certs\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.125215 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.125186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2242137b-d234-4c2f-ad8a-eb48e972dcef-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.125215 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.125218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-metrics-certs\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.125417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.125282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5s9\" (UniqueName: \"kubernetes.io/projected/2242137b-d234-4c2f-ad8a-eb48e972dcef-kube-api-access-kc5s9\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.125417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.125328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.125417 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:47.125412    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-serving-cert: secret "maas-api-e2e-inf-b-e39762-serving-cert" not found
Sep 08 15:30:47.125606 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:30:47.125489    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls podName:2242137b-d234-4c2f-ad8a-eb48e972dcef nodeName:}" failed. No retries permitted until 2026-09-08 15:30:47.625468799 +0000 UTC m=+796.700761070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls") pod "maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" (UID: "2242137b-d234-4c2f-ad8a-eb48e972dcef") : secret "maas-api-e2e-inf-b-e39762-serving-cert" not found
Sep 08 15:30:47.125949 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.125926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2242137b-d234-4c2f-ad8a-eb48e972dcef-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.127750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.127732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-metrics-certs\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.159467 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.159441    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5s9\" (UniqueName: \"kubernetes.io/projected/2242137b-d234-4c2f-ad8a-eb48e972dcef-kube-api-access-kc5s9\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.630939 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.630901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.633302 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.633280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls\") pod \"maas-api-e2e-inf-b-e39762-7574978f5f-l7q94\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:47.833289 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:47.833255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:48.141561 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.141540    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"]
Sep 08 15:30:48.144629 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:48.144550    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2242137b_d234_4c2f_ad8a_eb48e972dcef.slice/crio-2e445bea0c44783ff07e15e9cd87dca0ec3ee7ce48c580f5a933cca76a0dd9a0 WatchSource:0}: Error finding container 2e445bea0c44783ff07e15e9cd87dca0ec3ee7ce48c580f5a933cca76a0dd9a0: Status 404 returned error can't find the container with id 2e445bea0c44783ff07e15e9cd87dca0ec3ee7ce48c580f5a933cca76a0dd9a0
Sep 08 15:30:48.541177 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.541110    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"]
Sep 08 15:30:48.543168 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.543153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.549306 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.549288    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e39762-dockercfg-k2tpg\""
Sep 08 15:30:48.550371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.550354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e39762\""
Sep 08 15:30:48.577282 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.577259    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"]
Sep 08 15:30:48.640042 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.640007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jg6\" (UniqueName: \"kubernetes.io/projected/738e6dda-9a86-4dae-a495-a2880582cc75-kube-api-access-x5jg6\") pod \"payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.640170 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.640068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738e6dda-9a86-4dae-a495-a2880582cc75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.741178 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.741135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jg6\" (UniqueName: \"kubernetes.io/projected/738e6dda-9a86-4dae-a495-a2880582cc75-kube-api-access-x5jg6\") pod \"payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.741364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.741203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738e6dda-9a86-4dae-a495-a2880582cc75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.741961 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.741936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738e6dda-9a86-4dae-a495-a2880582cc75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.784589 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.784563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jg6\" (UniqueName: \"kubernetes.io/projected/738e6dda-9a86-4dae-a495-a2880582cc75-kube-api-access-x5jg6\") pod \"payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.853467 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.853407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:48.922266 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.922237    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" event={"ID":"2242137b-d234-4c2f-ad8a-eb48e972dcef","Type":"ContainerStarted","Data":"59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3"}
Sep 08 15:30:48.922391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.922306    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:48.922391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.922331    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" event={"ID":"2242137b-d234-4c2f-ad8a-eb48e972dcef","Type":"ContainerStarted","Data":"2e445bea0c44783ff07e15e9cd87dca0ec3ee7ce48c580f5a933cca76a0dd9a0"}
Sep 08 15:30:48.955885 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:48.955822    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" podStartSLOduration=2.552573192 podStartE2EDuration="2.955806679s" totalImagesPullingTime="403.233487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:48.954885454 +0000 UTC m=+798.030177771" watchObservedRunningTime="2026-09-08 15:30:48.955806679 +0000 UTC m=+798.031098971"
Sep 08 15:30:49.157282 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:49.157257    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"]
Sep 08 15:30:49.158612 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:49.158585    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738e6dda_9a86_4dae_a495_a2880582cc75.slice/crio-3b388ab485977958293df6f2e1e8a2b6fe52ff2be0552b14326ebf3fd83be49f WatchSource:0}: Error finding container 3b388ab485977958293df6f2e1e8a2b6fe52ff2be0552b14326ebf3fd83be49f: Status 404 returned error can't find the container with id 3b388ab485977958293df6f2e1e8a2b6fe52ff2be0552b14326ebf3fd83be49f
Sep 08 15:30:49.929130 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:49.929092    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw" event={"ID":"738e6dda-9a86-4dae-a495-a2880582cc75","Type":"ContainerStarted","Data":"ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99"}
Sep 08 15:30:49.929307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:49.929157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw" event={"ID":"738e6dda-9a86-4dae-a495-a2880582cc75","Type":"ContainerStarted","Data":"3b388ab485977958293df6f2e1e8a2b6fe52ff2be0552b14326ebf3fd83be49f"}
Sep 08 15:30:49.972674 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:49.972627    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw" podStartSLOduration=1.972612719 podStartE2EDuration="1.972612719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:49.966758207 +0000 UTC m=+799.042050519" watchObservedRunningTime="2026-09-08 15:30:49.972612719 +0000 UTC m=+799.047905003"
Sep 08 15:30:50.937034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:50.936993    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:30:54.935403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:54.935318    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:30:56.725044 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.725000    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:30:56.833728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.833695    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"]
Sep 08 15:30:56.839581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.837153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:56.844447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.844420    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8debe9\"/\"openshift-service-ca.crt\""
Sep 08 15:30:56.844447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.844444    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8debe9\"/\"autores-model-8debe9-kserve-self-signed-certs\""
Sep 08 15:30:56.844655 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.844445    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8debe9\"/\"kube-root-ca.crt\""
Sep 08 15:30:56.844655 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.844547    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8debe9\"/\"default-dockercfg-js68x\""
Sep 08 15:30:56.872476 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.872458    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"]
Sep 08 15:30:56.908209 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.908183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-tmp-dir\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:56.908317 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.908232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bnc\" (UniqueName: \"kubernetes.io/projected/6084691f-235b-4e31-9821-75150481f062-kube-api-access-j6bnc\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:56.908317 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.908291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-dshm\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:56.908384 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.908370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-home\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:56.908423 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.908397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084691f-235b-4e31-9821-75150481f062-tls-certs\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:56.908423 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:56.908420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-model-cache\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.009994 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.009907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-dshm\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-home\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084691f-235b-4e31-9821-75150481f062-tls-certs\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010284 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010215    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-model-cache\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010284 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-tmp-dir\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010400 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bnc\" (UniqueName: \"kubernetes.io/projected/6084691f-235b-4e31-9821-75150481f062-kube-api-access-j6bnc\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010627 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-home\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.010892 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.010868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-model-cache\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.011051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.011014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-tmp-dir\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.012758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.012738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-dshm\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.013125 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.013110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084691f-235b-4e31-9821-75150481f062-tls-certs\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.020034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.020006    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bnc\" (UniqueName: \"kubernetes.io/projected/6084691f-235b-4e31-9821-75150481f062-kube-api-access-j6bnc\") pod \"autores-model-8debe9-kserve-59c4c445c9-tntdb\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") " pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.150728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.150703    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:30:57.425320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.425295    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"]
Sep 08 15:30:57.427204 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:30:57.427180    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6084691f_235b_4e31_9821_75150481f062.slice/crio-14dae5bce3fe04c274783cd422d42c02075de058530d38f96d5f4f4eb7724ed6 WatchSource:0}: Error finding container 14dae5bce3fe04c274783cd422d42c02075de058530d38f96d5f4f4eb7724ed6: Status 404 returned error can't find the container with id 14dae5bce3fe04c274783cd422d42c02075de058530d38f96d5f4f4eb7724ed6
Sep 08 15:30:57.971301 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.971265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb" event={"ID":"6084691f-235b-4e31-9821-75150481f062","Type":"ContainerStarted","Data":"aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f"}
Sep 08 15:30:57.971691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:57.971312    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb" event={"ID":"6084691f-235b-4e31-9821-75150481f062","Type":"ContainerStarted","Data":"14dae5bce3fe04c274783cd422d42c02075de058530d38f96d5f4f4eb7724ed6"}
Sep 08 15:30:58.002333 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:30:58.002280    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb" podStartSLOduration=2.002259256 podStartE2EDuration="2.002259256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:58.00011045 +0000 UTC m=+807.075402744" watchObservedRunningTime="2026-09-08 15:30:58.002259256 +0000 UTC m=+807.077551549"
Sep 08 15:31:01.942345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:01.942313    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:31:02.748588 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:02.748552    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"]
Sep 08 15:31:02.748792 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:02.748760    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" podUID="ac5b175d-117b-43ef-a35b-12a36d0923df" containerName="maas-api" containerID="cri-o://300babfd8dd10c34adece1a08e69744d690f08fdb8751ccc410648885d98a9cf" gracePeriod=30
Sep 08 15:31:02.854118 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:02.854081    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"]
Sep 08 15:31:02.854319 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:02.854283    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp" podUID="38a4d313-44b3-4dfc-afd4-9a52f3e602c3" containerName="payload-pre-processing" containerID="cri-o://b68ad0a6a2580641b474f18155b04c0fc262db4e027217d646066088e53f02b7" gracePeriod=30
Sep 08 15:31:02.999970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:02.999938    2575 generic.go:356] "Generic (PLEG): container finished" podID="38a4d313-44b3-4dfc-afd4-9a52f3e602c3" containerID="b68ad0a6a2580641b474f18155b04c0fc262db4e027217d646066088e53f02b7" exitCode=0
Sep 08 15:31:03.000326 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.000025    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp" event={"ID":"38a4d313-44b3-4dfc-afd4-9a52f3e602c3","Type":"ContainerDied","Data":"b68ad0a6a2580641b474f18155b04c0fc262db4e027217d646066088e53f02b7"}
Sep 08 15:31:03.001439 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.001419    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac5b175d-117b-43ef-a35b-12a36d0923df" containerID="300babfd8dd10c34adece1a08e69744d690f08fdb8751ccc410648885d98a9cf" exitCode=0
Sep 08 15:31:03.001541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.001468    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" event={"ID":"ac5b175d-117b-43ef-a35b-12a36d0923df","Type":"ContainerDied","Data":"300babfd8dd10c34adece1a08e69744d690f08fdb8751ccc410648885d98a9cf"}
Sep 08 15:31:03.001541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.001509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt" event={"ID":"ac5b175d-117b-43ef-a35b-12a36d0923df","Type":"ContainerDied","Data":"a96c3b4954f494b0ee10102ea6e14e2e55ddd61a61d7936b6edf7a57b30985d2"}
Sep 08 15:31:03.001541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.001533    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a96c3b4954f494b0ee10102ea6e14e2e55ddd61a61d7936b6edf7a57b30985d2"
Sep 08 15:31:03.018837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.018820    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:31:03.111285 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.111263    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:31:03.161614 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.161586    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5b175d-117b-43ef-a35b-12a36d0923df-openshift-service-ca\") pod \"ac5b175d-117b-43ef-a35b-12a36d0923df\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") "
Sep 08 15:31:03.161742 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.161683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpsv4\" (UniqueName: \"kubernetes.io/projected/ac5b175d-117b-43ef-a35b-12a36d0923df-kube-api-access-jpsv4\") pod \"ac5b175d-117b-43ef-a35b-12a36d0923df\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") "
Sep 08 15:31:03.161790 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.161740    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls\") pod \"ac5b175d-117b-43ef-a35b-12a36d0923df\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") "
Sep 08 15:31:03.161790 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.161760    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-metrics-certs\") pod \"ac5b175d-117b-43ef-a35b-12a36d0923df\" (UID: \"ac5b175d-117b-43ef-a35b-12a36d0923df\") "
Sep 08 15:31:03.162012 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.161985    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5b175d-117b-43ef-a35b-12a36d0923df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac5b175d-117b-43ef-a35b-12a36d0923df" (UID: "ac5b175d-117b-43ef-a35b-12a36d0923df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:31:03.165002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.164974    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac5b175d-117b-43ef-a35b-12a36d0923df" (UID: "ac5b175d-117b-43ef-a35b-12a36d0923df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:31:03.165108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.165081    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5b175d-117b-43ef-a35b-12a36d0923df-kube-api-access-jpsv4" (OuterVolumeSpecName: "kube-api-access-jpsv4") pod "ac5b175d-117b-43ef-a35b-12a36d0923df" (UID: "ac5b175d-117b-43ef-a35b-12a36d0923df"). InnerVolumeSpecName "kube-api-access-jpsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:03.165108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.165082    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac5b175d-117b-43ef-a35b-12a36d0923df" (UID: "ac5b175d-117b-43ef-a35b-12a36d0923df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:31:03.262467 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262401    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl9k5\" (UniqueName: \"kubernetes.io/projected/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-kube-api-access-jl9k5\") pod \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") "
Sep 08 15:31:03.262467 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262440    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-plugins-config-volume\") pod \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\" (UID: \"38a4d313-44b3-4dfc-afd4-9a52f3e602c3\") "
Sep 08 15:31:03.262618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262594    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpsv4\" (UniqueName: \"kubernetes.io/projected/ac5b175d-117b-43ef-a35b-12a36d0923df-kube-api-access-jpsv4\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:03.262618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262606    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:03.262618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262614    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5b175d-117b-43ef-a35b-12a36d0923df-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:03.262731 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262622    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5b175d-117b-43ef-a35b-12a36d0923df-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:03.262802 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.262781    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38a4d313-44b3-4dfc-afd4-9a52f3e602c3" (UID: "38a4d313-44b3-4dfc-afd4-9a52f3e602c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:31:03.264563 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.264544    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-kube-api-access-jl9k5" (OuterVolumeSpecName: "kube-api-access-jl9k5") pod "38a4d313-44b3-4dfc-afd4-9a52f3e602c3" (UID: "38a4d313-44b3-4dfc-afd4-9a52f3e602c3"). InnerVolumeSpecName "kube-api-access-jl9k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:03.363269 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.363241    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl9k5\" (UniqueName: \"kubernetes.io/projected/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-kube-api-access-jl9k5\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:03.363269 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:03.363264    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4d313-44b3-4dfc-afd4-9a52f3e602c3-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:04.007493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.007464    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"
Sep 08 15:31:04.007493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.007495    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"
Sep 08 15:31:04.008005 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.007496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp" event={"ID":"38a4d313-44b3-4dfc-afd4-9a52f3e602c3","Type":"ContainerDied","Data":"4c1343b6031da3448f49969d897e382ab887e8a5d6e5b5c0b364d49b75fad743"}
Sep 08 15:31:04.008005 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.007557    2575 scope.go:117] "RemoveContainer" containerID="b68ad0a6a2580641b474f18155b04c0fc262db4e027217d646066088e53f02b7"
Sep 08 15:31:04.056377 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.056326    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"]
Sep 08 15:31:04.083438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.083415    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-425c6986-6f765bd8b6-l62lp"]
Sep 08 15:31:04.116597 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.116569    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"]
Sep 08 15:31:04.140360 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:04.140337    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-425c6986-686cfd77d-4bbvt"]
Sep 08 15:31:05.441095 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:05.441060    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a4d313-44b3-4dfc-afd4-9a52f3e602c3" path="/var/lib/kubelet/pods/38a4d313-44b3-4dfc-afd4-9a52f3e602c3/volumes"
Sep 08 15:31:05.441613 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:05.441587    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5b175d-117b-43ef-a35b-12a36d0923df" path="/var/lib/kubelet/pods/ac5b175d-117b-43ef-a35b-12a36d0923df/volumes"
Sep 08 15:31:07.150870 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:07.150795    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:31:07.151247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:07.150899    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:31:07.163669 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:07.163647    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:31:08.038170 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.038145    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:31:08.644695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.644647    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"]
Sep 08 15:31:08.645098 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.645070    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5b175d-117b-43ef-a35b-12a36d0923df" containerName="maas-api"
Sep 08 15:31:08.645098 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.645085    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5b175d-117b-43ef-a35b-12a36d0923df" containerName="maas-api"
Sep 08 15:31:08.645098 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.645096    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a4d313-44b3-4dfc-afd4-9a52f3e602c3" containerName="payload-pre-processing"
Sep 08 15:31:08.645247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.645102    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a4d313-44b3-4dfc-afd4-9a52f3e602c3" containerName="payload-pre-processing"
Sep 08 15:31:08.645247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.645168    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5b175d-117b-43ef-a35b-12a36d0923df" containerName="maas-api"
Sep 08 15:31:08.645247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.645177    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a4d313-44b3-4dfc-afd4-9a52f3e602c3" containerName="payload-pre-processing"
Sep 08 15:31:08.647987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.647969    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.651725 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.651702    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-96c53ecc-dockercfg-qzg8d\""
Sep 08 15:31:08.651855 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.651705    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-96c53ecc\""
Sep 08 15:31:08.666440 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.666419    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"]
Sep 08 15:31:08.812238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.812141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ec3ea-ecad-417d-b8a3-e98b24a08001-plugins-config-volume\") pod \"payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") " pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.812238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.812207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z79p\" (UniqueName: \"kubernetes.io/projected/067ec3ea-ecad-417d-b8a3-e98b24a08001-kube-api-access-9z79p\") pod \"payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") " pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.912981 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.912941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ec3ea-ecad-417d-b8a3-e98b24a08001-plugins-config-volume\") pod \"payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") " pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.913117 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.913001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z79p\" (UniqueName: \"kubernetes.io/projected/067ec3ea-ecad-417d-b8a3-e98b24a08001-kube-api-access-9z79p\") pod \"payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") " pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.913893 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.913871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ec3ea-ecad-417d-b8a3-e98b24a08001-plugins-config-volume\") pod \"payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") " pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.922915 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.922890    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z79p\" (UniqueName: \"kubernetes.io/projected/067ec3ea-ecad-417d-b8a3-e98b24a08001-kube-api-access-9z79p\") pod \"payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") " pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:08.957636 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:08.957613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:09.246613 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:09.246457    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"]
Sep 08 15:31:09.249267 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:31:09.249240    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067ec3ea_ecad_417d_b8a3_e98b24a08001.slice/crio-feab479ab01111bdcc7d184427d6f83941f0cec4f8ccd7f2870f0e248b08a0c8 WatchSource:0}: Error finding container feab479ab01111bdcc7d184427d6f83941f0cec4f8ccd7f2870f0e248b08a0c8: Status 404 returned error can't find the container with id feab479ab01111bdcc7d184427d6f83941f0cec4f8ccd7f2870f0e248b08a0c8
Sep 08 15:31:10.045116 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:10.045084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh" event={"ID":"067ec3ea-ecad-417d-b8a3-e98b24a08001","Type":"ContainerStarted","Data":"ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd"}
Sep 08 15:31:10.045567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:10.045139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh" event={"ID":"067ec3ea-ecad-417d-b8a3-e98b24a08001","Type":"ContainerStarted","Data":"feab479ab01111bdcc7d184427d6f83941f0cec4f8ccd7f2870f0e248b08a0c8"}
Sep 08 15:31:10.045567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:10.045187    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:10.076266 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:10.076224    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh" podStartSLOduration=2.076211189 podStartE2EDuration="2.076211189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:10.075711369 +0000 UTC m=+819.151003661" watchObservedRunningTime="2026-09-08 15:31:10.076211189 +0000 UTC m=+819.151503510"
Sep 08 15:31:17.559090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:17.559057    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:31:21.051689 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:21.051657    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:33.075631 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.075594    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"]
Sep 08 15:31:33.080150 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.080134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.087737 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.087716    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-d920a3f5-kserve-self-signed-certs\""
Sep 08 15:31:33.107413 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.107388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-dshm\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.107561 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.107443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-model-cache\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.107561 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.107524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-tmp-dir\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.107674 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.107650    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/400a65db-fa50-49a0-887d-787d8e51cbfc-tls-certs\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.107735 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.107687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s679w\" (UniqueName: \"kubernetes.io/projected/400a65db-fa50-49a0-887d-787d8e51cbfc-kube-api-access-s679w\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.107813 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.107782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-home\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.119005 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.118978    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"]
Sep 08 15:31:33.208188 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-home\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-dshm\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-model-cache\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-tmp-dir\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/400a65db-fa50-49a0-887d-787d8e51cbfc-tls-certs\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s679w\" (UniqueName: \"kubernetes.io/projected/400a65db-fa50-49a0-887d-787d8e51cbfc-kube-api-access-s679w\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208707 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208674    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-model-cache\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208871 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208684    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-home\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.208871 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.208750    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-tmp-dir\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.211255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.211231    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-dshm\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.211453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.211436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/400a65db-fa50-49a0-887d-787d8e51cbfc-tls-certs\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.223489 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.223470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s679w\" (UniqueName: \"kubernetes.io/projected/400a65db-fa50-49a0-887d-787d8e51cbfc-kube-api-access-s679w\") pod \"e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") " pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.389061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.389035    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:33.686109 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:33.686084    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"]
Sep 08 15:31:33.688224 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:31:33.688198    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod400a65db_fa50_49a0_887d_787d8e51cbfc.slice/crio-63f0c23a05f15ec36b33bba8d8f6f70457cb8f4e10aecac4517629d996cbcfd7 WatchSource:0}: Error finding container 63f0c23a05f15ec36b33bba8d8f6f70457cb8f4e10aecac4517629d996cbcfd7: Status 404 returned error can't find the container with id 63f0c23a05f15ec36b33bba8d8f6f70457cb8f4e10aecac4517629d996cbcfd7
Sep 08 15:31:34.171049 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:34.171017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn" event={"ID":"400a65db-fa50-49a0-887d-787d8e51cbfc","Type":"ContainerStarted","Data":"a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384"}
Sep 08 15:31:34.171459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:34.171062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn" event={"ID":"400a65db-fa50-49a0-887d-787d8e51cbfc","Type":"ContainerStarted","Data":"63f0c23a05f15ec36b33bba8d8f6f70457cb8f4e10aecac4517629d996cbcfd7"}
Sep 08 15:31:34.196766 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:34.196715    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn" podStartSLOduration=2.196700487 podStartE2EDuration="2.196700487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:34.196010972 +0000 UTC m=+843.271303263" watchObservedRunningTime="2026-09-08 15:31:34.196700487 +0000 UTC m=+843.271992780"
Sep 08 15:31:35.208303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:35.208266    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"]
Sep 08 15:31:38.405176 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:38.405132    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:31:39.644312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.644280    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"]
Sep 08 15:31:39.644811 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.644515    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh" podUID="067ec3ea-ecad-417d-b8a3-e98b24a08001" containerName="payload-processing" containerID="cri-o://ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd" gracePeriod=30
Sep 08 15:31:39.911116 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.911093    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:39.965768 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.965747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z79p\" (UniqueName: \"kubernetes.io/projected/067ec3ea-ecad-417d-b8a3-e98b24a08001-kube-api-access-9z79p\") pod \"067ec3ea-ecad-417d-b8a3-e98b24a08001\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") "
Sep 08 15:31:39.965916 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.965876    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ec3ea-ecad-417d-b8a3-e98b24a08001-plugins-config-volume\") pod \"067ec3ea-ecad-417d-b8a3-e98b24a08001\" (UID: \"067ec3ea-ecad-417d-b8a3-e98b24a08001\") "
Sep 08 15:31:39.966301 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.966273    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ec3ea-ecad-417d-b8a3-e98b24a08001-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "067ec3ea-ecad-417d-b8a3-e98b24a08001" (UID: "067ec3ea-ecad-417d-b8a3-e98b24a08001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:31:39.968112 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:39.968084    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ec3ea-ecad-417d-b8a3-e98b24a08001-kube-api-access-9z79p" (OuterVolumeSpecName: "kube-api-access-9z79p") pod "067ec3ea-ecad-417d-b8a3-e98b24a08001" (UID: "067ec3ea-ecad-417d-b8a3-e98b24a08001"). InnerVolumeSpecName "kube-api-access-9z79p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:40.067224 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.067191    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ec3ea-ecad-417d-b8a3-e98b24a08001-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:40.067224 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.067220    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z79p\" (UniqueName: \"kubernetes.io/projected/067ec3ea-ecad-417d-b8a3-e98b24a08001-kube-api-access-9z79p\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:40.203053 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.202983    2575 generic.go:356] "Generic (PLEG): container finished" podID="067ec3ea-ecad-417d-b8a3-e98b24a08001" containerID="ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd" exitCode=0
Sep 08 15:31:40.203053 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.203035    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"
Sep 08 15:31:40.203203 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.203071    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh" event={"ID":"067ec3ea-ecad-417d-b8a3-e98b24a08001","Type":"ContainerDied","Data":"ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd"}
Sep 08 15:31:40.203203 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.203119    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh" event={"ID":"067ec3ea-ecad-417d-b8a3-e98b24a08001","Type":"ContainerDied","Data":"feab479ab01111bdcc7d184427d6f83941f0cec4f8ccd7f2870f0e248b08a0c8"}
Sep 08 15:31:40.203203 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.203149    2575 scope.go:117] "RemoveContainer" containerID="ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd"
Sep 08 15:31:40.219739 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.219716    2575 scope.go:117] "RemoveContainer" containerID="ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd"
Sep 08 15:31:40.220137 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:31:40.220109    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd\": container with ID starting with ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd not found: ID does not exist" containerID="ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd"
Sep 08 15:31:40.220263 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.220138    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd"} err="failed to get container status \"ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd\": rpc error: code = NotFound desc = could not find container \"ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd\": container with ID starting with ad113c72775d9a25110570b0dab13db193ddc4c806b5e334436d5f856f2875cd not found: ID does not exist"
Sep 08 15:31:40.251358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.251334    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"]
Sep 08 15:31:40.261535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:40.261514    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-96c53ecc-6c96f9cf98-rlfjh"]
Sep 08 15:31:41.442033 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:41.442000    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067ec3ea-ecad-417d-b8a3-e98b24a08001" path="/var/lib/kubelet/pods/067ec3ea-ecad-417d-b8a3-e98b24a08001/volumes"
Sep 08 15:31:43.389345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:43.389315    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:43.389738 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:43.389360    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:43.402713 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:43.402685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:44.240057 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:44.240023    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:31:50.228953 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.228906    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb" podUID="6084691f-235b-4e31-9821-75150481f062" containerName="main" containerID="cri-o://aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f" gracePeriod=45
Sep 08 15:31:50.472156 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.472134    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:31:50.551586 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-model-cache\") pod \"6084691f-235b-4e31-9821-75150481f062\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") "
Sep 08 15:31:50.551691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551610    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-tmp-dir\") pod \"6084691f-235b-4e31-9821-75150481f062\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") "
Sep 08 15:31:50.551691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551638    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-home\") pod \"6084691f-235b-4e31-9821-75150481f062\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") "
Sep 08 15:31:50.551691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551676    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6bnc\" (UniqueName: \"kubernetes.io/projected/6084691f-235b-4e31-9821-75150481f062-kube-api-access-j6bnc\") pod \"6084691f-235b-4e31-9821-75150481f062\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") "
Sep 08 15:31:50.551812 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084691f-235b-4e31-9821-75150481f062-tls-certs\") pod \"6084691f-235b-4e31-9821-75150481f062\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") "
Sep 08 15:31:50.551812 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551746    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-model-cache" (OuterVolumeSpecName: "model-cache") pod "6084691f-235b-4e31-9821-75150481f062" (UID: "6084691f-235b-4e31-9821-75150481f062"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:31:50.551812 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-dshm\") pod \"6084691f-235b-4e31-9821-75150481f062\" (UID: \"6084691f-235b-4e31-9821-75150481f062\") "
Sep 08 15:31:50.552010 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.551988    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-home" (OuterVolumeSpecName: "home") pod "6084691f-235b-4e31-9821-75150481f062" (UID: "6084691f-235b-4e31-9821-75150481f062"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:31:50.552076 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.552002    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6084691f-235b-4e31-9821-75150481f062" (UID: "6084691f-235b-4e31-9821-75150481f062"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:31:50.552136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.552091    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:50.552136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.552109    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:50.552136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.552121    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:50.554118 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.554090    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-dshm" (OuterVolumeSpecName: "dshm") pod "6084691f-235b-4e31-9821-75150481f062" (UID: "6084691f-235b-4e31-9821-75150481f062"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:31:50.554214 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.554149    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6084691f-235b-4e31-9821-75150481f062-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6084691f-235b-4e31-9821-75150481f062" (UID: "6084691f-235b-4e31-9821-75150481f062"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:31:50.554253 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.554224    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6084691f-235b-4e31-9821-75150481f062-kube-api-access-j6bnc" (OuterVolumeSpecName: "kube-api-access-j6bnc") pod "6084691f-235b-4e31-9821-75150481f062" (UID: "6084691f-235b-4e31-9821-75150481f062"). InnerVolumeSpecName "kube-api-access-j6bnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:50.653095 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.653065    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bnc\" (UniqueName: \"kubernetes.io/projected/6084691f-235b-4e31-9821-75150481f062-kube-api-access-j6bnc\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:50.653095 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.653093    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084691f-235b-4e31-9821-75150481f062-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:50.653266 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:50.653104    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084691f-235b-4e31-9821-75150481f062-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:31:51.267031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.266998    2575 generic.go:356] "Generic (PLEG): container finished" podID="6084691f-235b-4e31-9821-75150481f062" containerID="aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f" exitCode=0
Sep 08 15:31:51.267493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.267049    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"
Sep 08 15:31:51.267493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.267060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb" event={"ID":"6084691f-235b-4e31-9821-75150481f062","Type":"ContainerDied","Data":"aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f"}
Sep 08 15:31:51.267493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.267101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb" event={"ID":"6084691f-235b-4e31-9821-75150481f062","Type":"ContainerDied","Data":"14dae5bce3fe04c274783cd422d42c02075de058530d38f96d5f4f4eb7724ed6"}
Sep 08 15:31:51.267493 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.267124    2575 scope.go:117] "RemoveContainer" containerID="aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f"
Sep 08 15:31:51.279036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.279013    2575 scope.go:117] "RemoveContainer" containerID="aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f"
Sep 08 15:31:51.279351 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:31:51.279329    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f\": container with ID starting with aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f not found: ID does not exist" containerID="aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f"
Sep 08 15:31:51.279433 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.279361    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f"} err="failed to get container status \"aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f\": rpc error: code = NotFound desc = could not find container \"aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f\": container with ID starting with aec0d2a833b080c72e5dcaabae25902bda8e3ecb70f345c9c1da001e143e5f2f not found: ID does not exist"
Sep 08 15:31:51.311893 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.311868    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"]
Sep 08 15:31:51.322767 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.322747    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-8debe9/autores-model-8debe9-kserve-59c4c445c9-tntdb"]
Sep 08 15:31:51.441450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:51.441428    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6084691f-235b-4e31-9821-75150481f062" path="/var/lib/kubelet/pods/6084691f-235b-4e31-9821-75150481f062/volumes"
Sep 08 15:31:55.249216 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:55.249182    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"]
Sep 08 15:31:55.249573 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:31:55.249373    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy" containerID="cri-o://337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c" gracePeriod=30
Sep 08 15:32:00.498555 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.498529    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:32:00.600044 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.599960    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"]
Sep 08 15:32:00.600382 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600363    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ec3ea-ecad-417d-b8a3-e98b24a08001" containerName="payload-processing"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600384    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ec3ea-ecad-417d-b8a3-e98b24a08001" containerName="payload-processing"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600405    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6084691f-235b-4e31-9821-75150481f062" containerName="main"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600413    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6084691f-235b-4e31-9821-75150481f062" containerName="main"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600428    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600436    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600462    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy"
Sep 08 15:32:00.600483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600470    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy"
Sep 08 15:32:00.600820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600547    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ec3ea-ecad-417d-b8a3-e98b24a08001" containerName="payload-processing"
Sep 08 15:32:00.600820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600563    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy"
Sep 08 15:32:00.600820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600575    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c57828-a594-46d4-8b8e-91f6da343297" containerName="istio-proxy"
Sep 08 15:32:00.600820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.600584    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6084691f-235b-4e31-9821-75150481f062" containerName="main"
Sep 08 15:32:00.603417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.603397    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.606660 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.606637    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-356350-serving-cert\""
Sep 08 15:32:00.616271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.616247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"]
Sep 08 15:32:00.636287 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636264    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-data\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636303    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rstl\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-kube-api-access-2rstl\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636325    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-socket\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636345    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c57828-a594-46d4-8b8e-91f6da343297-istio-podinfo\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636373    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-certs\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636397    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c57828-a594-46d4-8b8e-91f6da343297-istiod-ca-cert\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636458    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-istio-token\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636486    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-envoy\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636518    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-credential-socket\") pod \"31c57828-a594-46d4-8b8e-91f6da343297\" (UID: \"31c57828-a594-46d4-8b8e-91f6da343297\") "
Sep 08 15:32:00.636663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636574    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-data" (OuterVolumeSpecName: "istio-data") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:00.636663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636582    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:00.636968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636811    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.636968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636816    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:00.636968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636828    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-data\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.636968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.636830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c57828-a594-46d4-8b8e-91f6da343297-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:32:00.637176 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.637062    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:00.638939 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.638910    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-istio-token" (OuterVolumeSpecName: "istio-token") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:32:00.639375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.639345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-kube-api-access-2rstl" (OuterVolumeSpecName: "kube-api-access-2rstl") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "kube-api-access-2rstl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:32:00.639498 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.639470    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:00.639560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.639525    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31c57828-a594-46d4-8b8e-91f6da343297-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31c57828-a594-46d4-8b8e-91f6da343297" (UID: "31c57828-a594-46d4-8b8e-91f6da343297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:32:00.738014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.737982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-openshift-service-ca\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.738137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-maas-api-tls\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.738137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvbg\" (UniqueName: \"kubernetes.io/projected/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-kube-api-access-sqvbg\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.738242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-metrics-certs\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.738278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738243    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rstl\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-kube-api-access-2rstl\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.738278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738258    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c57828-a594-46d4-8b8e-91f6da343297-istio-podinfo\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.738278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738267    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-workload-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.738278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738275    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c57828-a594-46d4-8b8e-91f6da343297-istiod-ca-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.738397 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738282    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c57828-a594-46d4-8b8e-91f6da343297-istio-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.738397 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738291    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-istio-envoy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.738397 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.738298    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c57828-a594-46d4-8b8e-91f6da343297-credential-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:00.838750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.838724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-openshift-service-ca\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.838921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.838763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-maas-api-tls\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.838921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.838785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvbg\" (UniqueName: \"kubernetes.io/projected/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-kube-api-access-sqvbg\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.838921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.838806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-metrics-certs\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.839441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.839419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-openshift-service-ca\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.841404 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.841382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-metrics-certs\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.841513 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.841461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-maas-api-tls\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.847491 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.847468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvbg\" (UniqueName: \"kubernetes.io/projected/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-kube-api-access-sqvbg\") pod \"maas-api-e2e-explicit-356350-744bd94f65-cshrl\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:00.914170 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:00.914148    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:01.145288 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.145259    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:32:01.197664 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.197642    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"]
Sep 08 15:32:01.199355 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:32:01.199333    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b23c8e_bb79_4fb3_95df_ed5e7a62a4a3.slice/crio-c989881c9745eba1a6b10b231a8233968b6bccfe581b784bedcd7d001c2f685d WatchSource:0}: Error finding container c989881c9745eba1a6b10b231a8233968b6bccfe581b784bedcd7d001c2f685d: Status 404 returned error can't find the container with id c989881c9745eba1a6b10b231a8233968b6bccfe581b784bedcd7d001c2f685d
Sep 08 15:32:01.320945 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.320919    2575 generic.go:356] "Generic (PLEG): container finished" podID="31c57828-a594-46d4-8b8e-91f6da343297" containerID="337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c" exitCode=0
Sep 08 15:32:01.321071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.320967    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"
Sep 08 15:32:01.321071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.320998    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" event={"ID":"31c57828-a594-46d4-8b8e-91f6da343297","Type":"ContainerDied","Data":"337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c"}
Sep 08 15:32:01.321071 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.321050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz" event={"ID":"31c57828-a594-46d4-8b8e-91f6da343297","Type":"ContainerDied","Data":"3fa825a6f16c21df8f23df7f59e2025d066f91c75e407b10a031d15ed93f3303"}
Sep 08 15:32:01.321221 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.321073    2575 scope.go:117] "RemoveContainer" containerID="337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c"
Sep 08 15:32:01.322300 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.322277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl" event={"ID":"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3","Type":"ContainerStarted","Data":"c989881c9745eba1a6b10b231a8233968b6bccfe581b784bedcd7d001c2f685d"}
Sep 08 15:32:01.329812 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.329794    2575 scope.go:117] "RemoveContainer" containerID="837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"
Sep 08 15:32:01.337659 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.337644    2575 scope.go:117] "RemoveContainer" containerID="337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c"
Sep 08 15:32:01.337908 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:32:01.337888    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c\": container with ID starting with 337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c not found: ID does not exist" containerID="337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c"
Sep 08 15:32:01.337961 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.337916    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c"} err="failed to get container status \"337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c\": rpc error: code = NotFound desc = could not find container \"337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c\": container with ID starting with 337b6f1fe62ccf233980f2b16f31bf15c92e87f8d683bd2bd953662947fc048c not found: ID does not exist"
Sep 08 15:32:01.337961 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.337931    2575 scope.go:117] "RemoveContainer" containerID="837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"
Sep 08 15:32:01.338202 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:32:01.338185    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c\": container with ID starting with 837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c not found: ID does not exist" containerID="837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"
Sep 08 15:32:01.338242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.338210    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c"} err="failed to get container status \"837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c\": rpc error: code = NotFound desc = could not find container \"837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c\": container with ID starting with 837b0cf75c7afb28a4087ba97350993bd4e99f7b6b0c47195b42e9b3ae28297c not found: ID does not exist"
Sep 08 15:32:01.355709 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.355679    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"]
Sep 08 15:32:01.365227 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.365206    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8debe9-openshift-default-796d556b76-6jlnz"]
Sep 08 15:32:01.441309 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.441288    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c57828-a594-46d4-8b8e-91f6da343297" path="/var/lib/kubelet/pods/31c57828-a594-46d4-8b8e-91f6da343297/volumes"
Sep 08 15:32:01.624567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.624487    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"]
Sep 08 15:32:01.629181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.629162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.633269 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.633249    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-356350\""
Sep 08 15:32:01.633410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.633395    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-356350-dockercfg-7wjpb\""
Sep 08 15:32:01.647197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.647175    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"]
Sep 08 15:32:01.745689 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.745662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-plugins-config-volume\") pod \"payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") " pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.745789 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.745699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27pn\" (UniqueName: \"kubernetes.io/projected/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-kube-api-access-f27pn\") pod \"payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") " pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.846178 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.846157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-plugins-config-volume\") pod \"payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") " pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.846261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.846192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27pn\" (UniqueName: \"kubernetes.io/projected/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-kube-api-access-f27pn\") pod \"payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") " pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.846761 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.846745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-plugins-config-volume\") pod \"payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") " pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.857366 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.857342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27pn\" (UniqueName: \"kubernetes.io/projected/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-kube-api-access-f27pn\") pod \"payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") " pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:01.939357 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:01.939328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:02.213961 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.213813    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"]
Sep 08 15:32:02.216539 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:32:02.216502    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeec10a1_6ab0_4e9e_8fbd_324f06c462de.slice/crio-3c52b12788bc966c37cb6a9ffc64a04bc2b497d12e9b095477938e9ae8271fa3 WatchSource:0}: Error finding container 3c52b12788bc966c37cb6a9ffc64a04bc2b497d12e9b095477938e9ae8271fa3: Status 404 returned error can't find the container with id 3c52b12788bc966c37cb6a9ffc64a04bc2b497d12e9b095477938e9ae8271fa3
Sep 08 15:32:02.329562 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.329529    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl" event={"ID":"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3","Type":"ContainerStarted","Data":"0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8"}
Sep 08 15:32:02.329754 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.329711    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:02.331348 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.331328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f" event={"ID":"aeec10a1-6ab0-4e9e-8fbd-324f06c462de","Type":"ContainerStarted","Data":"fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344"}
Sep 08 15:32:02.331443 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.331365    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f" event={"ID":"aeec10a1-6ab0-4e9e-8fbd-324f06c462de","Type":"ContainerStarted","Data":"3c52b12788bc966c37cb6a9ffc64a04bc2b497d12e9b095477938e9ae8271fa3"}
Sep 08 15:32:02.331443 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.331391    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:02.387278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.387235    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f" podStartSLOduration=1.387220679 podStartE2EDuration="1.387220679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:02.385883709 +0000 UTC m=+871.461176024" watchObservedRunningTime="2026-09-08 15:32:02.387220679 +0000 UTC m=+871.462512971"
Sep 08 15:32:02.387714 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:02.387688    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl" podStartSLOduration=1.79564478 podStartE2EDuration="2.387679837s" totalImagesPullingTime="592.035057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:02.359761188 +0000 UTC m=+871.435053491" watchObservedRunningTime="2026-09-08 15:32:02.387679837 +0000 UTC m=+871.462972127"
Sep 08 15:32:08.342964 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:08.342936    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:32:13.339032 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:13.338999    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:32:18.009854 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:18.009807    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:32:30.700217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:30.700124    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"]
Sep 08 15:32:31.771892 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:31.771804    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"]
Sep 08 15:32:40.634805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:40.634756    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:32:45.732110 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:45.732070    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn" podUID="400a65db-fa50-49a0-887d-787d8e51cbfc" containerName="main" containerID="cri-o://a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384" gracePeriod=45
Sep 08 15:32:45.987787 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:45.987735    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:32:46.005598 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005573    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-model-cache\") pod \"400a65db-fa50-49a0-887d-787d8e51cbfc\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") "
Sep 08 15:32:46.005730 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005635    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/400a65db-fa50-49a0-887d-787d8e51cbfc-tls-certs\") pod \"400a65db-fa50-49a0-887d-787d8e51cbfc\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") "
Sep 08 15:32:46.005730 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005682    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-home\") pod \"400a65db-fa50-49a0-887d-787d8e51cbfc\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") "
Sep 08 15:32:46.005880 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005762    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-dshm\") pod \"400a65db-fa50-49a0-887d-787d8e51cbfc\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") "
Sep 08 15:32:46.005880 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005804    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s679w\" (UniqueName: \"kubernetes.io/projected/400a65db-fa50-49a0-887d-787d8e51cbfc-kube-api-access-s679w\") pod \"400a65db-fa50-49a0-887d-787d8e51cbfc\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") "
Sep 08 15:32:46.005880 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005874    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-tmp-dir\") pod \"400a65db-fa50-49a0-887d-787d8e51cbfc\" (UID: \"400a65db-fa50-49a0-887d-787d8e51cbfc\") "
Sep 08 15:32:46.006042 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.005874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "400a65db-fa50-49a0-887d-787d8e51cbfc" (UID: "400a65db-fa50-49a0-887d-787d8e51cbfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:46.006210 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.006180    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:46.006399 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.006358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-home" (OuterVolumeSpecName: "home") pod "400a65db-fa50-49a0-887d-787d8e51cbfc" (UID: "400a65db-fa50-49a0-887d-787d8e51cbfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:46.006735 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.006543    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "400a65db-fa50-49a0-887d-787d8e51cbfc" (UID: "400a65db-fa50-49a0-887d-787d8e51cbfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:46.008436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.008413    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a65db-fa50-49a0-887d-787d8e51cbfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "400a65db-fa50-49a0-887d-787d8e51cbfc" (UID: "400a65db-fa50-49a0-887d-787d8e51cbfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:32:46.008529 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.008468    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-dshm" (OuterVolumeSpecName: "dshm") pod "400a65db-fa50-49a0-887d-787d8e51cbfc" (UID: "400a65db-fa50-49a0-887d-787d8e51cbfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:46.008642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.008623    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400a65db-fa50-49a0-887d-787d8e51cbfc-kube-api-access-s679w" (OuterVolumeSpecName: "kube-api-access-s679w") pod "400a65db-fa50-49a0-887d-787d8e51cbfc" (UID: "400a65db-fa50-49a0-887d-787d8e51cbfc"). InnerVolumeSpecName "kube-api-access-s679w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:32:46.106777 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.106755    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:46.106777 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.106774    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s679w\" (UniqueName: \"kubernetes.io/projected/400a65db-fa50-49a0-887d-787d8e51cbfc-kube-api-access-s679w\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:46.106936 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.106785    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:46.106936 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.106794    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/400a65db-fa50-49a0-887d-787d8e51cbfc-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:46.106936 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.106803    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/400a65db-fa50-49a0-887d-787d8e51cbfc-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:46.566185 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.566153    2575 generic.go:356] "Generic (PLEG): container finished" podID="400a65db-fa50-49a0-887d-787d8e51cbfc" containerID="a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384" exitCode=0
Sep 08 15:32:46.566369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.566213    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"
Sep 08 15:32:46.566369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.566253    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn" event={"ID":"400a65db-fa50-49a0-887d-787d8e51cbfc","Type":"ContainerDied","Data":"a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384"}
Sep 08 15:32:46.566369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.566303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn" event={"ID":"400a65db-fa50-49a0-887d-787d8e51cbfc","Type":"ContainerDied","Data":"63f0c23a05f15ec36b33bba8d8f6f70457cb8f4e10aecac4517629d996cbcfd7"}
Sep 08 15:32:46.566369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.566322    2575 scope.go:117] "RemoveContainer" containerID="a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384"
Sep 08 15:32:46.576141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.576125    2575 scope.go:117] "RemoveContainer" containerID="a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384"
Sep 08 15:32:46.576432 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:32:46.576415    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384\": container with ID starting with a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384 not found: ID does not exist" containerID="a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384"
Sep 08 15:32:46.576484 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.576439    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384"} err="failed to get container status \"a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384\": rpc error: code = NotFound desc = could not find container \"a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384\": container with ID starting with a12768b83c3a0017e9ff9cc15d7b5b7b4910c1fc20836e02b07772d569b09384 not found: ID does not exist"
Sep 08 15:32:46.590553 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.590526    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"]
Sep 08 15:32:46.596736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.596714    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-d920a3f5-kserve-54bbbd9c4c-crsqn"]
Sep 08 15:32:46.791594 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:46.791561    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7" podUID="d687e584-155e-409c-ab9d-97a446791cd7" containerName="main" containerID="cri-o://2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b" gracePeriod=45
Sep 08 15:32:47.034728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.034705    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:32:47.115983 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.115961    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d687e584-155e-409c-ab9d-97a446791cd7-tls-certs\") pod \"d687e584-155e-409c-ab9d-97a446791cd7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") "
Sep 08 15:32:47.116099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.115999    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsxj8\" (UniqueName: \"kubernetes.io/projected/d687e584-155e-409c-ab9d-97a446791cd7-kube-api-access-xsxj8\") pod \"d687e584-155e-409c-ab9d-97a446791cd7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") "
Sep 08 15:32:47.116099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-home\") pod \"d687e584-155e-409c-ab9d-97a446791cd7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") "
Sep 08 15:32:47.116099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116081    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-model-cache\") pod \"d687e584-155e-409c-ab9d-97a446791cd7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") "
Sep 08 15:32:47.116230 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-tmp-dir\") pod \"d687e584-155e-409c-ab9d-97a446791cd7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") "
Sep 08 15:32:47.116230 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116169    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-dshm\") pod \"d687e584-155e-409c-ab9d-97a446791cd7\" (UID: \"d687e584-155e-409c-ab9d-97a446791cd7\") "
Sep 08 15:32:47.116361 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116332    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d687e584-155e-409c-ab9d-97a446791cd7" (UID: "d687e584-155e-409c-ab9d-97a446791cd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:47.116424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116347    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-home" (OuterVolumeSpecName: "home") pod "d687e584-155e-409c-ab9d-97a446791cd7" (UID: "d687e584-155e-409c-ab9d-97a446791cd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:47.116476 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116433    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d687e584-155e-409c-ab9d-97a446791cd7" (UID: "d687e584-155e-409c-ab9d-97a446791cd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:47.116533 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116474    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:47.116533 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.116494    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:47.118242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.118217    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d687e584-155e-409c-ab9d-97a446791cd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d687e584-155e-409c-ab9d-97a446791cd7" (UID: "d687e584-155e-409c-ab9d-97a446791cd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:32:47.118334 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.118291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d687e584-155e-409c-ab9d-97a446791cd7-kube-api-access-xsxj8" (OuterVolumeSpecName: "kube-api-access-xsxj8") pod "d687e584-155e-409c-ab9d-97a446791cd7" (UID: "d687e584-155e-409c-ab9d-97a446791cd7"). InnerVolumeSpecName "kube-api-access-xsxj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:32:47.118334 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.118305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-dshm" (OuterVolumeSpecName: "dshm") pod "d687e584-155e-409c-ab9d-97a446791cd7" (UID: "d687e584-155e-409c-ab9d-97a446791cd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:32:47.217691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.217626    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d687e584-155e-409c-ab9d-97a446791cd7-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:47.217691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.217656    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxj8\" (UniqueName: \"kubernetes.io/projected/d687e584-155e-409c-ab9d-97a446791cd7-kube-api-access-xsxj8\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:47.217691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.217684    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:47.217691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.217693    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d687e584-155e-409c-ab9d-97a446791cd7-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:32:47.442182 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.442153    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400a65db-fa50-49a0-887d-787d8e51cbfc" path="/var/lib/kubelet/pods/400a65db-fa50-49a0-887d-787d8e51cbfc/volumes"
Sep 08 15:32:47.572740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.572647    2575 generic.go:356] "Generic (PLEG): container finished" podID="d687e584-155e-409c-ab9d-97a446791cd7" containerID="2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b" exitCode=0
Sep 08 15:32:47.572740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.572686    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"
Sep 08 15:32:47.572962 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.572734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7" event={"ID":"d687e584-155e-409c-ab9d-97a446791cd7","Type":"ContainerDied","Data":"2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b"}
Sep 08 15:32:47.572962 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.572791    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7" event={"ID":"d687e584-155e-409c-ab9d-97a446791cd7","Type":"ContainerDied","Data":"114cabfddc77a278d061052d6bfef28f37541842555bbc4a9e8efb5246586868"}
Sep 08 15:32:47.572962 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.572817    2575 scope.go:117] "RemoveContainer" containerID="2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b"
Sep 08 15:32:47.580833 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.580811    2575 scope.go:117] "RemoveContainer" containerID="2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b"
Sep 08 15:32:47.581153 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:32:47.581135    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b\": container with ID starting with 2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b not found: ID does not exist" containerID="2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b"
Sep 08 15:32:47.581211 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.581160    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b"} err="failed to get container status \"2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b\": rpc error: code = NotFound desc = could not find container \"2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b\": container with ID starting with 2d9748e05efa070f7cc55deb519a1e7d36df7fed47b626fe29fe85a7d840421b not found: ID does not exist"
Sep 08 15:32:47.616402 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.616383    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"]
Sep 08 15:32:47.620719 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:47.620700    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-d920a3f5-kserve-6b5f99d46f-879w7"]
Sep 08 15:32:49.441798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:49.441766    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d687e584-155e-409c-ab9d-97a446791cd7" path="/var/lib/kubelet/pods/d687e584-155e-409c-ab9d-97a446791cd7/volumes"
Sep 08 15:32:58.107887 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:32:58.106610    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:33:11.902335 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902300    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"]
Sep 08 15:33:11.902891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902664    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d687e584-155e-409c-ab9d-97a446791cd7" containerName="main"
Sep 08 15:33:11.902891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902676    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d687e584-155e-409c-ab9d-97a446791cd7" containerName="main"
Sep 08 15:33:11.902891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902694    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400a65db-fa50-49a0-887d-787d8e51cbfc" containerName="main"
Sep 08 15:33:11.902891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902700    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="400a65db-fa50-49a0-887d-787d8e51cbfc" containerName="main"
Sep 08 15:33:11.902891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902752    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d687e584-155e-409c-ab9d-97a446791cd7" containerName="main"
Sep 08 15:33:11.902891 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.902761    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="400a65db-fa50-49a0-887d-787d8e51cbfc" containerName="main"
Sep 08 15:33:11.905684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.905665    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:11.910410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.910384    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-356350\"/\"openshift-service-ca.crt\""
Sep 08 15:33:11.910560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.910410    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-356350\"/\"explicit-model-356350-kserve-self-signed-certs\""
Sep 08 15:33:11.910560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.910420    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-356350\"/\"default-dockercfg-4fm8z\""
Sep 08 15:33:11.910560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.910427    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-356350\"/\"kube-root-ca.crt\""
Sep 08 15:33:11.916717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:11.916691    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"]
Sep 08 15:33:12.007189 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.007162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-model-cache\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.007310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.007204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-dshm\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.007310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.007225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2ht\" (UniqueName: \"kubernetes.io/projected/a735b837-8df1-4a81-87ce-a3d73e6e7c12-kube-api-access-xt2ht\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.007403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.007305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tls-certs\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.007403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.007353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-home\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.007403 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.007371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tmp-dir\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.108509 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-home\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.108509 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tmp-dir\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.108681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-model-cache\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.108681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-dshm\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.108681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2ht\" (UniqueName: \"kubernetes.io/projected/a735b837-8df1-4a81-87ce-a3d73e6e7c12-kube-api-access-xt2ht\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.108811 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tls-certs\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.109004 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.108986    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-home\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.109050 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.109027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tmp-dir\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.109086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.109054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-model-cache\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.111193 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.111172    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-dshm\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.111521 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.111499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tls-certs\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.118322 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.118299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2ht\" (UniqueName: \"kubernetes.io/projected/a735b837-8df1-4a81-87ce-a3d73e6e7c12-kube-api-access-xt2ht\") pod \"explicit-model-356350-kserve-78f768466f-vt47v\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") " pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.216035 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.215974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:12.491249 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.491219    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"]
Sep 08 15:33:12.493133 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:33:12.493111    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda735b837_8df1_4a81_87ce_a3d73e6e7c12.slice/crio-15f4efae6567b2bd2b0def84b901edac10954f24e78485976b3614ffc5b88deb WatchSource:0}: Error finding container 15f4efae6567b2bd2b0def84b901edac10954f24e78485976b3614ffc5b88deb: Status 404 returned error can't find the container with id 15f4efae6567b2bd2b0def84b901edac10954f24e78485976b3614ffc5b88deb
Sep 08 15:33:12.691029 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.690996    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v" event={"ID":"a735b837-8df1-4a81-87ce-a3d73e6e7c12","Type":"ContainerStarted","Data":"f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f"}
Sep 08 15:33:12.691174 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.691039    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v" event={"ID":"a735b837-8df1-4a81-87ce-a3d73e6e7c12","Type":"ContainerStarted","Data":"15f4efae6567b2bd2b0def84b901edac10954f24e78485976b3614ffc5b88deb"}
Sep 08 15:33:12.711273 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:12.711227    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v" podStartSLOduration=1.7112128580000001 podStartE2EDuration="1.711212858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:12.710575362 +0000 UTC m=+941.785867666" watchObservedRunningTime="2026-09-08 15:33:12.711212858 +0000 UTC m=+941.786505151"
Sep 08 15:33:15.610771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:15.610741    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:33:22.216964 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:22.216923    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:22.217388 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:22.216990    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:22.229833 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:22.229809    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:22.749362 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:22.749334    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:29.698483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.698445    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"]
Sep 08 15:33:29.701609 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.701588    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.704692 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.704670    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e39762\"/\"kube-root-ca.crt\""
Sep 08 15:33:29.706259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.706237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e39762\"/\"openshift-service-ca.crt\""
Sep 08 15:33:29.706361 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.706272    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e39762\"/\"default-dockercfg-sjplf\""
Sep 08 15:33:29.706361 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.706269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e39762\"/\"test-model-e39762-kserve-self-signed-certs\""
Sep 08 15:33:29.710782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.710758    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"]
Sep 08 15:33:29.849444 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.849416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8pz\" (UniqueName: \"kubernetes.io/projected/e2b8156b-a8b8-414e-afec-8996aebbf706-kube-api-access-kx8pz\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.849543 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.849462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-model-cache\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.849543 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.849479    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-tmp-dir\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.849543 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.849530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b8156b-a8b8-414e-afec-8996aebbf706-tls-certs\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.849662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.849557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-home\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.849662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.849600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-dshm\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.950900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.950821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8pz\" (UniqueName: \"kubernetes.io/projected/e2b8156b-a8b8-414e-afec-8996aebbf706-kube-api-access-kx8pz\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.950900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.950889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-model-cache\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.950907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-tmp-dir\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.950923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b8156b-a8b8-414e-afec-8996aebbf706-tls-certs\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.950939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-home\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.950982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-dshm\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.951341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-tmp-dir\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.951360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-model-cache\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.951471 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.951395    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-home\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.953616 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.953592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-dshm\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.953805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.953785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b8156b-a8b8-414e-afec-8996aebbf706-tls-certs\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:29.960485 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:29.960466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8pz\" (UniqueName: \"kubernetes.io/projected/e2b8156b-a8b8-414e-afec-8996aebbf706-kube-api-access-kx8pz\") pod \"test-model-e39762-kserve-5fc5fbfd65-9rmvg\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") " pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:30.013515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:30.013490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:30.292325 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:30.292302    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"]
Sep 08 15:33:30.294624 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:33:30.294599    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2b8156b_a8b8_414e_afec_8996aebbf706.slice/crio-91f88c693773ee1db90595d658046deee56557acc72b483edbd4ba841b70f962 WatchSource:0}: Error finding container 91f88c693773ee1db90595d658046deee56557acc72b483edbd4ba841b70f962: Status 404 returned error can't find the container with id 91f88c693773ee1db90595d658046deee56557acc72b483edbd4ba841b70f962
Sep 08 15:33:30.777154 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:30.777119    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg" event={"ID":"e2b8156b-a8b8-414e-afec-8996aebbf706","Type":"ContainerStarted","Data":"38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216"}
Sep 08 15:33:30.777540 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:30.777165    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg" event={"ID":"e2b8156b-a8b8-414e-afec-8996aebbf706","Type":"ContainerStarted","Data":"91f88c693773ee1db90595d658046deee56557acc72b483edbd4ba841b70f962"}
Sep 08 15:33:30.801069 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:30.801023    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg" podStartSLOduration=1.8010083620000001 podStartE2EDuration="1.801008362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:30.799666331 +0000 UTC m=+959.874958646" watchObservedRunningTime="2026-09-08 15:33:30.801008362 +0000 UTC m=+959.876300655"
Sep 08 15:33:31.995936 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:31.995906    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"]
Sep 08 15:33:32.225385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:32.225350    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:33:40.014579 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:40.014535    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:40.015072 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:40.014604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:40.027934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:40.027904    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:40.834089 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:40.834057    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:33:42.778973 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:42.778938    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"]
Sep 08 15:33:42.779542 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:42.779145    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl" podUID="b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" containerName="maas-api" containerID="cri-o://0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8" gracePeriod=30
Sep 08 15:33:42.823447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:42.823413    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"]
Sep 08 15:33:42.823718 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:42.823663    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f" podUID="aeec10a1-6ab0-4e9e-8fbd-324f06c462de" containerName="payload-processing" containerID="cri-o://fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344" gracePeriod=30
Sep 08 15:33:43.075644 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.075618    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:33:43.100605 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.100578    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:33:43.265107 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265082    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-openshift-service-ca\") pod \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") "
Sep 08 15:33:43.265107 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265111    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqvbg\" (UniqueName: \"kubernetes.io/projected/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-kube-api-access-sqvbg\") pod \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") "
Sep 08 15:33:43.265320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-maas-api-tls\") pod \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") "
Sep 08 15:33:43.265320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265198    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-plugins-config-volume\") pod \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") "
Sep 08 15:33:43.265320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265231    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f27pn\" (UniqueName: \"kubernetes.io/projected/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-kube-api-access-f27pn\") pod \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\" (UID: \"aeec10a1-6ab0-4e9e-8fbd-324f06c462de\") "
Sep 08 15:33:43.265320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265259    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-metrics-certs\") pod \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\" (UID: \"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3\") "
Sep 08 15:33:43.265554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265526    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" (UID: "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:33:43.265661 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.265624    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aeec10a1-6ab0-4e9e-8fbd-324f06c462de" (UID: "aeec10a1-6ab0-4e9e-8fbd-324f06c462de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:33:43.267374 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.267346    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" (UID: "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:33:43.267532 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.267513    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" (UID: "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:33:43.267649 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.267635    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-kube-api-access-sqvbg" (OuterVolumeSpecName: "kube-api-access-sqvbg") pod "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" (UID: "b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3"). InnerVolumeSpecName "kube-api-access-sqvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:33:43.267762 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.267738    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-kube-api-access-f27pn" (OuterVolumeSpecName: "kube-api-access-f27pn") pod "aeec10a1-6ab0-4e9e-8fbd-324f06c462de" (UID: "aeec10a1-6ab0-4e9e-8fbd-324f06c462de"). InnerVolumeSpecName "kube-api-access-f27pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:33:43.366796 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.366741    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f27pn\" (UniqueName: \"kubernetes.io/projected/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-kube-api-access-f27pn\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.366796 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.366763    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.366796 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.366772    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.366796 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.366780    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvbg\" (UniqueName: \"kubernetes.io/projected/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-kube-api-access-sqvbg\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.366796 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.366790    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.366796 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.366798    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeec10a1-6ab0-4e9e-8fbd-324f06c462de-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.840535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.840499    2575 generic.go:356] "Generic (PLEG): container finished" podID="b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" containerID="0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8" exitCode=0
Sep 08 15:33:43.840958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.840547    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"
Sep 08 15:33:43.840958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.840582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl" event={"ID":"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3","Type":"ContainerDied","Data":"0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8"}
Sep 08 15:33:43.840958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.840640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl" event={"ID":"b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3","Type":"ContainerDied","Data":"c989881c9745eba1a6b10b231a8233968b6bccfe581b784bedcd7d001c2f685d"}
Sep 08 15:33:43.840958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.840675    2575 scope.go:117] "RemoveContainer" containerID="0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8"
Sep 08 15:33:43.842222 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.842200    2575 generic.go:356] "Generic (PLEG): container finished" podID="aeec10a1-6ab0-4e9e-8fbd-324f06c462de" containerID="fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344" exitCode=0
Sep 08 15:33:43.842294 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.842245    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"
Sep 08 15:33:43.842294 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.842261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f" event={"ID":"aeec10a1-6ab0-4e9e-8fbd-324f06c462de","Type":"ContainerDied","Data":"fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344"}
Sep 08 15:33:43.842407 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.842298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f" event={"ID":"aeec10a1-6ab0-4e9e-8fbd-324f06c462de","Type":"ContainerDied","Data":"3c52b12788bc966c37cb6a9ffc64a04bc2b497d12e9b095477938e9ae8271fa3"}
Sep 08 15:33:43.849784 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.849767    2575 scope.go:117] "RemoveContainer" containerID="0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8"
Sep 08 15:33:43.850079 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:33:43.850058    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8\": container with ID starting with 0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8 not found: ID does not exist" containerID="0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8"
Sep 08 15:33:43.850166 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.850086    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8"} err="failed to get container status \"0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8\": rpc error: code = NotFound desc = could not find container \"0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8\": container with ID starting with 0fddd280ec0ea4319ee7c45148412784998fd16f86c0fbb20bfb774125b776c8 not found: ID does not exist"
Sep 08 15:33:43.850166 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.850102    2575 scope.go:117] "RemoveContainer" containerID="fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344"
Sep 08 15:33:43.857981 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.857965    2575 scope.go:117] "RemoveContainer" containerID="fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344"
Sep 08 15:33:43.858349 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:33:43.858323    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344\": container with ID starting with fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344 not found: ID does not exist" containerID="fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344"
Sep 08 15:33:43.858465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.858355    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344"} err="failed to get container status \"fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344\": rpc error: code = NotFound desc = could not find container \"fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344\": container with ID starting with fee957a1337a7937036f66be87da1d6aac2b0c4164e456daf9eb90a6a81b2344 not found: ID does not exist"
Sep 08 15:33:43.860441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.860421    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"]
Sep 08 15:33:43.864581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.864557    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-356350-744bd94f65-cshrl"]
Sep 08 15:33:43.877684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.877660    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"]
Sep 08 15:33:43.882722 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:43.882702    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-356350-55598b5b7c-s4n5f"]
Sep 08 15:33:45.442532 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:45.442495    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeec10a1-6ab0-4e9e-8fbd-324f06c462de" path="/var/lib/kubelet/pods/aeec10a1-6ab0-4e9e-8fbd-324f06c462de/volumes"
Sep 08 15:33:45.443148 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:45.442885    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" path="/var/lib/kubelet/pods/b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3/volumes"
Sep 08 15:33:46.705863 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:46.705802    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:33:47.015558 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.015481    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v" podUID="a735b837-8df1-4a81-87ce-a3d73e6e7c12" containerName="main" containerID="cri-o://f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f" gracePeriod=45
Sep 08 15:33:47.261451 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.261427    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:47.296907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.296829    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tmp-dir\") pod \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") "
Sep 08 15:33:47.296907 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.296901    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-model-cache\") pod \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") "
Sep 08 15:33:47.297109 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.296946    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-dshm\") pod \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") "
Sep 08 15:33:47.297109 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.296974    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt2ht\" (UniqueName: \"kubernetes.io/projected/a735b837-8df1-4a81-87ce-a3d73e6e7c12-kube-api-access-xt2ht\") pod \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") "
Sep 08 15:33:47.297109 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.297015    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-home\") pod \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") "
Sep 08 15:33:47.297109 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.297092    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a735b837-8df1-4a81-87ce-a3d73e6e7c12" (UID: "a735b837-8df1-4a81-87ce-a3d73e6e7c12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:47.297320 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.297194    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-model-cache" (OuterVolumeSpecName: "model-cache") pod "a735b837-8df1-4a81-87ce-a3d73e6e7c12" (UID: "a735b837-8df1-4a81-87ce-a3d73e6e7c12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:47.297379 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.297333    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-home" (OuterVolumeSpecName: "home") pod "a735b837-8df1-4a81-87ce-a3d73e6e7c12" (UID: "a735b837-8df1-4a81-87ce-a3d73e6e7c12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:47.297379 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.297356    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:47.297379 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.297374    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:47.299876 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.299820    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-dshm" (OuterVolumeSpecName: "dshm") pod "a735b837-8df1-4a81-87ce-a3d73e6e7c12" (UID: "a735b837-8df1-4a81-87ce-a3d73e6e7c12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:47.300984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.300956    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a735b837-8df1-4a81-87ce-a3d73e6e7c12-kube-api-access-xt2ht" (OuterVolumeSpecName: "kube-api-access-xt2ht") pod "a735b837-8df1-4a81-87ce-a3d73e6e7c12" (UID: "a735b837-8df1-4a81-87ce-a3d73e6e7c12"). InnerVolumeSpecName "kube-api-access-xt2ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:33:47.398517 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.398492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tls-certs\") pod \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\" (UID: \"a735b837-8df1-4a81-87ce-a3d73e6e7c12\") "
Sep 08 15:33:47.398802 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.398785    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:47.398877 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.398811    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2ht\" (UniqueName: \"kubernetes.io/projected/a735b837-8df1-4a81-87ce-a3d73e6e7c12-kube-api-access-xt2ht\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:47.398877 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.398829    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a735b837-8df1-4a81-87ce-a3d73e6e7c12-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:47.400643 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.400626    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a735b837-8df1-4a81-87ce-a3d73e6e7c12" (UID: "a735b837-8df1-4a81-87ce-a3d73e6e7c12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:33:47.500247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.500220    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a735b837-8df1-4a81-87ce-a3d73e6e7c12-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:33:47.864808 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.864767    2575 generic.go:356] "Generic (PLEG): container finished" podID="a735b837-8df1-4a81-87ce-a3d73e6e7c12" containerID="f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f" exitCode=0
Sep 08 15:33:47.864808 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.864810    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"
Sep 08 15:33:47.865342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.864840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v" event={"ID":"a735b837-8df1-4a81-87ce-a3d73e6e7c12","Type":"ContainerDied","Data":"f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f"}
Sep 08 15:33:47.865342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.864896    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v" event={"ID":"a735b837-8df1-4a81-87ce-a3d73e6e7c12","Type":"ContainerDied","Data":"15f4efae6567b2bd2b0def84b901edac10954f24e78485976b3614ffc5b88deb"}
Sep 08 15:33:47.865342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.864916    2575 scope.go:117] "RemoveContainer" containerID="f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f"
Sep 08 15:33:47.874163 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.874139    2575 scope.go:117] "RemoveContainer" containerID="f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f"
Sep 08 15:33:47.874472 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:33:47.874449    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f\": container with ID starting with f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f not found: ID does not exist" containerID="f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f"
Sep 08 15:33:47.874598 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.874476    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f"} err="failed to get container status \"f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f\": rpc error: code = NotFound desc = could not find container \"f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f\": container with ID starting with f6695af2758596ebdd09a4a8877e257983b7124140df2065c890bd41517aae9f not found: ID does not exist"
Sep 08 15:33:47.913797 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.913768    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"]
Sep 08 15:33:47.915181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:47.915160    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-356350/explicit-model-356350-kserve-78f768466f-vt47v"]
Sep 08 15:33:49.441750 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:33:49.441718    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a735b837-8df1-4a81-87ce-a3d73e6e7c12" path="/var/lib/kubelet/pods/a735b837-8df1-4a81-87ce-a3d73e6e7c12/volumes"
Sep 08 15:34:15.912124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912092    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"]
Sep 08 15:34:15.912715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912555    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a735b837-8df1-4a81-87ce-a3d73e6e7c12" containerName="main"
Sep 08 15:34:15.912715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912575    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a735b837-8df1-4a81-87ce-a3d73e6e7c12" containerName="main"
Sep 08 15:34:15.912715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912607    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeec10a1-6ab0-4e9e-8fbd-324f06c462de" containerName="payload-processing"
Sep 08 15:34:15.912715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912616    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeec10a1-6ab0-4e9e-8fbd-324f06c462de" containerName="payload-processing"
Sep 08 15:34:15.912715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912637    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" containerName="maas-api"
Sep 08 15:34:15.912715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912646    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" containerName="maas-api"
Sep 08 15:34:15.913062 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912728    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeec10a1-6ab0-4e9e-8fbd-324f06c462de" containerName="payload-processing"
Sep 08 15:34:15.913062 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912738    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a735b837-8df1-4a81-87ce-a3d73e6e7c12" containerName="main"
Sep 08 15:34:15.913062 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.912751    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b23c8e-bb79-4fb3-95df-ed5e7a62a4a3" containerName="maas-api"
Sep 08 15:34:15.915662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.915642    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:15.920054 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.920033    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-86d73c\"/\"openshift-service-ca.crt\""
Sep 08 15:34:15.920164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.920069    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-86d73c\"/\"default-dockercfg-6g4mb\""
Sep 08 15:34:15.920164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.920078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-86d73c\"/\"noten-model-86d73c-kserve-self-signed-certs\""
Sep 08 15:34:15.920164 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.920089    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-86d73c\"/\"kube-root-ca.crt\""
Sep 08 15:34:15.927316 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:15.927295    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"]
Sep 08 15:34:16.022592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.022563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-model-cache\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.022763 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.022600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-tmp-dir\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.022763 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.022622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-dshm\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.022763 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.022733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-home\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.022970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.022769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlpr\" (UniqueName: \"kubernetes.io/projected/c7de9878-6f56-4475-a72b-7e2b4508d95e-kube-api-access-rqlpr\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.022970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.022875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7de9878-6f56-4475-a72b-7e2b4508d95e-tls-certs\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.123587 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.123558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlpr\" (UniqueName: \"kubernetes.io/projected/c7de9878-6f56-4475-a72b-7e2b4508d95e-kube-api-access-rqlpr\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.123717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.123603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7de9878-6f56-4475-a72b-7e2b4508d95e-tls-certs\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.123717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.123665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-model-cache\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.123717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.123712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-tmp-dir\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.123921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.123745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-dshm\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.123921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.123802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-home\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.124145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.124114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-model-cache\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.124278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.124161    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-tmp-dir\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.124278 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.124185    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-home\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.126200 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.126180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-dshm\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.126446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.126428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7de9878-6f56-4475-a72b-7e2b4508d95e-tls-certs\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.132804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.132780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlpr\" (UniqueName: \"kubernetes.io/projected/c7de9878-6f56-4475-a72b-7e2b4508d95e-kube-api-access-rqlpr\") pod \"noten-model-86d73c-kserve-66cc6897f5-4qrs2\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") " pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.224676 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.224605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:16.509830 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:16.509791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"]
Sep 08 15:34:16.512795 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:34:16.512762    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7de9878_6f56_4475_a72b_7e2b4508d95e.slice/crio-b6c7ab510c7c8b29858eb381c24a96a1b517efc91c0005c594466a7b5fc1042b WatchSource:0}: Error finding container b6c7ab510c7c8b29858eb381c24a96a1b517efc91c0005c594466a7b5fc1042b: Status 404 returned error can't find the container with id b6c7ab510c7c8b29858eb381c24a96a1b517efc91c0005c594466a7b5fc1042b
Sep 08 15:34:17.017148 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:17.017115    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2" event={"ID":"c7de9878-6f56-4475-a72b-7e2b4508d95e","Type":"ContainerStarted","Data":"7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663"}
Sep 08 15:34:17.017148 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:17.017153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2" event={"ID":"c7de9878-6f56-4475-a72b-7e2b4508d95e","Type":"ContainerStarted","Data":"b6c7ab510c7c8b29858eb381c24a96a1b517efc91c0005c594466a7b5fc1042b"}
Sep 08 15:34:17.039903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:17.039860    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2" podStartSLOduration=2.039827335 podStartE2EDuration="2.039827335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:17.038534688 +0000 UTC m=+1006.113826996" watchObservedRunningTime="2026-09-08 15:34:17.039827335 +0000 UTC m=+1006.115119659"
Sep 08 15:34:22.006295 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:22.006262    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:34:26.123302 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:26.123269    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"]
Sep 08 15:34:26.225504 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:26.225466    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:37.109902 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.109870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:34:37.523364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.523328    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"]
Sep 08 15:34:37.528076 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.528057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.531150 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.531129    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9fbfa439\""
Sep 08 15:34:37.539536 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.539513    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"]
Sep 08 15:34:37.547918 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.547896    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"]
Sep 08 15:34:37.556718 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.556695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.560695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.560670    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9fbfa439-dockercfg-576mt\""
Sep 08 15:34:37.571246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.571227    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"]
Sep 08 15:34:37.594304 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.594282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88z4\" (UniqueName: \"kubernetes.io/projected/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-kube-api-access-x88z4\") pod \"payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.594397 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.594316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.695698 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.695664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5pwc\" (UniqueName: \"kubernetes.io/projected/6055c843-2195-4409-ab3d-f38fd63fc281-kube-api-access-b5pwc\") pod \"payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") " pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.695884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.695725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6055c843-2195-4409-ab3d-f38fd63fc281-plugins-config-volume\") pod \"payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") " pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.695884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.695765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88z4\" (UniqueName: \"kubernetes.io/projected/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-kube-api-access-x88z4\") pod \"payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.695884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.695871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.696568 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.696543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.705037 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.705019    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88z4\" (UniqueName: \"kubernetes.io/projected/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-kube-api-access-x88z4\") pod \"payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.796783 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.796721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5pwc\" (UniqueName: \"kubernetes.io/projected/6055c843-2195-4409-ab3d-f38fd63fc281-kube-api-access-b5pwc\") pod \"payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") " pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.796889 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.796784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6055c843-2195-4409-ab3d-f38fd63fc281-plugins-config-volume\") pod \"payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") " pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.797465 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.797447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6055c843-2195-4409-ab3d-f38fd63fc281-plugins-config-volume\") pod \"payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") " pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.806255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.806237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5pwc\" (UniqueName: \"kubernetes.io/projected/6055c843-2195-4409-ab3d-f38fd63fc281-kube-api-access-b5pwc\") pod \"payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") " pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:37.837610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.837592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:37.867984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:37.867960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:38.110794 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:38.110766    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"]
Sep 08 15:34:38.112371 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:34:38.112344    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fba4bc7_37bc_40f4_8368_bf9d5a4f6ffd.slice/crio-195f5e9c33507ce1608fe5167b8d5570e5be8d9f8ef14d83b2362aeb19f9b63a WatchSource:0}: Error finding container 195f5e9c33507ce1608fe5167b8d5570e5be8d9f8ef14d83b2362aeb19f9b63a: Status 404 returned error can't find the container with id 195f5e9c33507ce1608fe5167b8d5570e5be8d9f8ef14d83b2362aeb19f9b63a
Sep 08 15:34:38.118166 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:38.118141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc" event={"ID":"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd","Type":"ContainerStarted","Data":"195f5e9c33507ce1608fe5167b8d5570e5be8d9f8ef14d83b2362aeb19f9b63a"}
Sep 08 15:34:38.152892 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:38.152872    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"]
Sep 08 15:34:38.163802 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:34:38.163776    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6055c843_2195_4409_ab3d_f38fd63fc281.slice/crio-ffe3b420ec3b73e51fe869c05e87df0dc07b7ea70709440a03c8f4a4a2c64b64 WatchSource:0}: Error finding container ffe3b420ec3b73e51fe869c05e87df0dc07b7ea70709440a03c8f4a4a2c64b64: Status 404 returned error can't find the container with id ffe3b420ec3b73e51fe869c05e87df0dc07b7ea70709440a03c8f4a4a2c64b64
Sep 08 15:34:39.124629 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.124590    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r" event={"ID":"6055c843-2195-4409-ab3d-f38fd63fc281","Type":"ContainerStarted","Data":"b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9"}
Sep 08 15:34:39.125090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.124658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r" event={"ID":"6055c843-2195-4409-ab3d-f38fd63fc281","Type":"ContainerStarted","Data":"ffe3b420ec3b73e51fe869c05e87df0dc07b7ea70709440a03c8f4a4a2c64b64"}
Sep 08 15:34:39.125090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.124693    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:39.126197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.126178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc" event={"ID":"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd","Type":"ContainerStarted","Data":"f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390"}
Sep 08 15:34:39.126297 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.126286    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:39.148010 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.147959    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r" podStartSLOduration=2.1479443480000002 podStartE2EDuration="2.147944348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:39.145881937 +0000 UTC m=+1028.221174230" watchObservedRunningTime="2026-09-08 15:34:39.147944348 +0000 UTC m=+1028.223236644"
Sep 08 15:34:39.168511 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.168470    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc" podStartSLOduration=2.168458277 podStartE2EDuration="2.168458277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:39.165785793 +0000 UTC m=+1028.241078104" watchObservedRunningTime="2026-09-08 15:34:39.168458277 +0000 UTC m=+1028.243750613"
Sep 08 15:34:39.578028 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:39.577999    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"]
Sep 08 15:34:41.144516 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.144475    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2" podUID="c7de9878-6f56-4475-a72b-7e2b4508d95e" containerName="main" containerID="cri-o://7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663" gracePeriod=45
Sep 08 15:34:41.397477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.397427    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:41.528826 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.528801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-home\") pod \"c7de9878-6f56-4475-a72b-7e2b4508d95e\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") "
Sep 08 15:34:41.528964 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.528835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7de9878-6f56-4475-a72b-7e2b4508d95e-tls-certs\") pod \"c7de9878-6f56-4475-a72b-7e2b4508d95e\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") "
Sep 08 15:34:41.528964 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.528868    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-tmp-dir\") pod \"c7de9878-6f56-4475-a72b-7e2b4508d95e\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") "
Sep 08 15:34:41.528964 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.528941    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-dshm\") pod \"c7de9878-6f56-4475-a72b-7e2b4508d95e\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") "
Sep 08 15:34:41.529125 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.528992    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-model-cache\") pod \"c7de9878-6f56-4475-a72b-7e2b4508d95e\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") "
Sep 08 15:34:41.529125 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.529029    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqlpr\" (UniqueName: \"kubernetes.io/projected/c7de9878-6f56-4475-a72b-7e2b4508d95e-kube-api-access-rqlpr\") pod \"c7de9878-6f56-4475-a72b-7e2b4508d95e\" (UID: \"c7de9878-6f56-4475-a72b-7e2b4508d95e\") "
Sep 08 15:34:41.529125 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.529100    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-home" (OuterVolumeSpecName: "home") pod "c7de9878-6f56-4475-a72b-7e2b4508d95e" (UID: "c7de9878-6f56-4475-a72b-7e2b4508d95e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:41.529288 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.529167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7de9878-6f56-4475-a72b-7e2b4508d95e" (UID: "c7de9878-6f56-4475-a72b-7e2b4508d95e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:41.529352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.529286    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7de9878-6f56-4475-a72b-7e2b4508d95e" (UID: "c7de9878-6f56-4475-a72b-7e2b4508d95e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:41.529352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.529307    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:41.529352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.529319    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:41.531267 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.531242    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7de9878-6f56-4475-a72b-7e2b4508d95e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7de9878-6f56-4475-a72b-7e2b4508d95e" (UID: "c7de9878-6f56-4475-a72b-7e2b4508d95e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:34:41.531380 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.531271    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-dshm" (OuterVolumeSpecName: "dshm") pod "c7de9878-6f56-4475-a72b-7e2b4508d95e" (UID: "c7de9878-6f56-4475-a72b-7e2b4508d95e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:41.531380 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.531314    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7de9878-6f56-4475-a72b-7e2b4508d95e-kube-api-access-rqlpr" (OuterVolumeSpecName: "kube-api-access-rqlpr") pod "c7de9878-6f56-4475-a72b-7e2b4508d95e" (UID: "c7de9878-6f56-4475-a72b-7e2b4508d95e"). InnerVolumeSpecName "kube-api-access-rqlpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:41.629799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.629775    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7de9878-6f56-4475-a72b-7e2b4508d95e-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:41.629799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.629801    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:41.629945 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.629810    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7de9878-6f56-4475-a72b-7e2b4508d95e-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:41.629945 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:41.629819    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlpr\" (UniqueName: \"kubernetes.io/projected/c7de9878-6f56-4475-a72b-7e2b4508d95e-kube-api-access-rqlpr\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:42.146211 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.146169    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7de9878-6f56-4475-a72b-7e2b4508d95e" containerID="7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663" exitCode=0
Sep 08 15:34:42.146634 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.146227    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"
Sep 08 15:34:42.146634 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.146246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2" event={"ID":"c7de9878-6f56-4475-a72b-7e2b4508d95e","Type":"ContainerDied","Data":"7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663"}
Sep 08 15:34:42.146634 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.146283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2" event={"ID":"c7de9878-6f56-4475-a72b-7e2b4508d95e","Type":"ContainerDied","Data":"b6c7ab510c7c8b29858eb381c24a96a1b517efc91c0005c594466a7b5fc1042b"}
Sep 08 15:34:42.146634 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.146300    2575 scope.go:117] "RemoveContainer" containerID="7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663"
Sep 08 15:34:42.158193 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.158172    2575 scope.go:117] "RemoveContainer" containerID="7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663"
Sep 08 15:34:42.158494 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:34:42.158475    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663\": container with ID starting with 7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663 not found: ID does not exist" containerID="7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663"
Sep 08 15:34:42.158545 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.158504    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663"} err="failed to get container status \"7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663\": rpc error: code = NotFound desc = could not find container \"7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663\": container with ID starting with 7d5619e1480cbc05307ffa85363f489dc9f59d3adaa9617032e6a5069ae13663 not found: ID does not exist"
Sep 08 15:34:42.190784 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.190756    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"]
Sep 08 15:34:42.197224 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:42.197200    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-86d73c/noten-model-86d73c-kserve-66cc6897f5-4qrs2"]
Sep 08 15:34:43.441076 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:43.441040    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7de9878-6f56-4475-a72b-7e2b4508d95e" path="/var/lib/kubelet/pods/c7de9878-6f56-4475-a72b-7e2b4508d95e/volumes"
Sep 08 15:34:49.126666 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:49.126619    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:34:50.135029 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:50.134998    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:34:50.135497 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:50.135066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:34:54.597488 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.597421    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg" podUID="e2b8156b-a8b8-414e-afec-8996aebbf706" containerName="main" containerID="cri-o://38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216" gracePeriod=45
Sep 08 15:34:54.849535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.849484    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:34:54.939283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939254    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-home\") pod \"e2b8156b-a8b8-414e-afec-8996aebbf706\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") "
Sep 08 15:34:54.939417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939287    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-model-cache\") pod \"e2b8156b-a8b8-414e-afec-8996aebbf706\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") "
Sep 08 15:34:54.939417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939311    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx8pz\" (UniqueName: \"kubernetes.io/projected/e2b8156b-a8b8-414e-afec-8996aebbf706-kube-api-access-kx8pz\") pod \"e2b8156b-a8b8-414e-afec-8996aebbf706\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") "
Sep 08 15:34:54.939417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939397    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-tmp-dir\") pod \"e2b8156b-a8b8-414e-afec-8996aebbf706\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") "
Sep 08 15:34:54.939595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939416    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b8156b-a8b8-414e-afec-8996aebbf706-tls-certs\") pod \"e2b8156b-a8b8-414e-afec-8996aebbf706\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") "
Sep 08 15:34:54.939595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-dshm\") pod \"e2b8156b-a8b8-414e-afec-8996aebbf706\" (UID: \"e2b8156b-a8b8-414e-afec-8996aebbf706\") "
Sep 08 15:34:54.939595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939537    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-home" (OuterVolumeSpecName: "home") pod "e2b8156b-a8b8-414e-afec-8996aebbf706" (UID: "e2b8156b-a8b8-414e-afec-8996aebbf706"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:54.939595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939551    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2b8156b-a8b8-414e-afec-8996aebbf706" (UID: "e2b8156b-a8b8-414e-afec-8996aebbf706"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:54.939815 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939699    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2b8156b-a8b8-414e-afec-8996aebbf706" (UID: "e2b8156b-a8b8-414e-afec-8996aebbf706"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:54.939903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939835    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:54.939903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939871    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:54.939903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.939890    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:54.941805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.941782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b8156b-a8b8-414e-afec-8996aebbf706-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2b8156b-a8b8-414e-afec-8996aebbf706" (UID: "e2b8156b-a8b8-414e-afec-8996aebbf706"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:34:54.941805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.941785    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b8156b-a8b8-414e-afec-8996aebbf706-kube-api-access-kx8pz" (OuterVolumeSpecName: "kube-api-access-kx8pz") pod "e2b8156b-a8b8-414e-afec-8996aebbf706" (UID: "e2b8156b-a8b8-414e-afec-8996aebbf706"). InnerVolumeSpecName "kube-api-access-kx8pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:54.941945 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:54.941793    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-dshm" (OuterVolumeSpecName: "dshm") pod "e2b8156b-a8b8-414e-afec-8996aebbf706" (UID: "e2b8156b-a8b8-414e-afec-8996aebbf706"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:55.041036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.041002    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8pz\" (UniqueName: \"kubernetes.io/projected/e2b8156b-a8b8-414e-afec-8996aebbf706-kube-api-access-kx8pz\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:55.041036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.041032    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b8156b-a8b8-414e-afec-8996aebbf706-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:55.041036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.041042    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b8156b-a8b8-414e-afec-8996aebbf706-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:34:55.215988 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.215962    2575 generic.go:356] "Generic (PLEG): container finished" podID="e2b8156b-a8b8-414e-afec-8996aebbf706" containerID="38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216" exitCode=0
Sep 08 15:34:55.216090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.215999    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg" event={"ID":"e2b8156b-a8b8-414e-afec-8996aebbf706","Type":"ContainerDied","Data":"38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216"}
Sep 08 15:34:55.216090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.216047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg" event={"ID":"e2b8156b-a8b8-414e-afec-8996aebbf706","Type":"ContainerDied","Data":"91f88c693773ee1db90595d658046deee56557acc72b483edbd4ba841b70f962"}
Sep 08 15:34:55.216090 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.216066    2575 scope.go:117] "RemoveContainer" containerID="38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216"
Sep 08 15:34:55.216209 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.216014    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"
Sep 08 15:34:55.225415 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.225399    2575 scope.go:117] "RemoveContainer" containerID="38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216"
Sep 08 15:34:55.225732 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:34:55.225713    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216\": container with ID starting with 38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216 not found: ID does not exist" containerID="38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216"
Sep 08 15:34:55.225793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.225742    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216"} err="failed to get container status \"38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216\": rpc error: code = NotFound desc = could not find container \"38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216\": container with ID starting with 38cd7555ff020eaccd52efb71400d6eddb1ffaabe4cfe513eab36c9ec4225216 not found: ID does not exist"
Sep 08 15:34:55.238000 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.237976    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"]
Sep 08 15:34:55.242677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.242656    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e39762/test-model-e39762-kserve-5fc5fbfd65-9rmvg"]
Sep 08 15:34:55.442094 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:55.442070    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b8156b-a8b8-414e-afec-8996aebbf706" path="/var/lib/kubelet/pods/e2b8156b-a8b8-414e-afec-8996aebbf706/volumes"
Sep 08 15:34:58.115277 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:58.115246    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"]
Sep 08 15:34:58.115664 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:34:58.115469    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" containerID="cri-o://ae3db3c218b985b0ffa37d8e27084fdddbfcf81e1867188cf7ec69abdd656400" gracePeriod=30
Sep 08 15:35:00.501771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:00.501732    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:35:00.571768 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:00.571729    2575 patch_prober.go:28] interesting pod/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 15:35:00.571953 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:00.571797    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 15:35:03.258461 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.258430    2575 generic.go:356] "Generic (PLEG): container finished" podID="c2116d54-3f72-4a7e-960d-7582e423136b" containerID="ae3db3c218b985b0ffa37d8e27084fdddbfcf81e1867188cf7ec69abdd656400" exitCode=0
Sep 08 15:35:03.258884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.258505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" event={"ID":"c2116d54-3f72-4a7e-960d-7582e423136b","Type":"ContainerDied","Data":"ae3db3c218b985b0ffa37d8e27084fdddbfcf81e1867188cf7ec69abdd656400"}
Sep 08 15:35:03.258884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.258580    2575 scope.go:117] "RemoveContainer" containerID="01d1cfbf11d9604ae7b5b8c3e6bccd0eb0f63fd854681a2a1a95c48a378e063c"
Sep 08 15:35:03.366518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.366492    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:35:03.506930 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.506903    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-certs\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.506954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-socket\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.506996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-data\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507032    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2116d54-3f72-4a7e-960d-7582e423136b-istio-podinfo\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507070    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-envoy\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507140    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-credential-socket\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507144    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:35:03.507271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507178    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trhxk\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-kube-api-access-trhxk\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2116d54-3f72-4a7e-960d-7582e423136b-istiod-ca-cert\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507237    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-istio-token\") pod \"c2116d54-3f72-4a7e-960d-7582e423136b\" (UID: \"c2116d54-3f72-4a7e-960d-7582e423136b\") "
Sep 08 15:35:03.507531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507273    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:35:03.507531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507282    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:35:03.507531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507470    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:35:03.507684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507577    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.507684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507597    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-workload-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.507684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507611    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-data\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.507684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507624    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-credential-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.507887 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.507866    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2116d54-3f72-4a7e-960d-7582e423136b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:03.509608 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.509585    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:03.509751 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.509729    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:35:03.510115 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.510095    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2116d54-3f72-4a7e-960d-7582e423136b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:35:03.510180 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.510133    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-kube-api-access-trhxk" (OuterVolumeSpecName: "kube-api-access-trhxk") pod "c2116d54-3f72-4a7e-960d-7582e423136b" (UID: "c2116d54-3f72-4a7e-960d-7582e423136b"). InnerVolumeSpecName "kube-api-access-trhxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:03.610535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.609490    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trhxk\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-kube-api-access-trhxk\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.610535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.609520    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2116d54-3f72-4a7e-960d-7582e423136b-istiod-ca-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.610535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.609534    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2116d54-3f72-4a7e-960d-7582e423136b-istio-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.610535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.609548    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2116d54-3f72-4a7e-960d-7582e423136b-istio-podinfo\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:03.610535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:03.609566    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2116d54-3f72-4a7e-960d-7582e423136b-istio-envoy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:04.264970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:04.264933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4" event={"ID":"c2116d54-3f72-4a7e-960d-7582e423136b","Type":"ContainerDied","Data":"f87be1a13b72305575375decb3b2d6e3c2676af47419b5866dafe86c49fda12f"}
Sep 08 15:35:04.264970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:04.264954    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"
Sep 08 15:35:04.265447 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:04.265017    2575 scope.go:117] "RemoveContainer" containerID="ae3db3c218b985b0ffa37d8e27084fdddbfcf81e1867188cf7ec69abdd656400"
Sep 08 15:35:04.288572 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:04.288549    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"]
Sep 08 15:35:04.293684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:04.293663    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-34fed8-openshift-default-7746975b7d-4knw4"]
Sep 08 15:35:05.440789 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:05.440749    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" path="/var/lib/kubelet/pods/c2116d54-3f72-4a7e-960d-7582e423136b/volumes"
Sep 08 15:35:08.944560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:08.944527    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"]
Sep 08 15:35:08.945102 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:08.944757    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" podUID="2242137b-d234-4c2f-ad8a-eb48e972dcef" containerName="maas-api" containerID="cri-o://59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3" gracePeriod=30
Sep 08 15:35:09.038047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.038019    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"]
Sep 08 15:35:09.038295 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.038268    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw" podUID="738e6dda-9a86-4dae-a495-a2880582cc75" containerName="payload-processing" containerID="cri-o://ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99" gracePeriod=30
Sep 08 15:35:09.201506 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.201451    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:35:09.249510 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.249480    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls\") pod \"2242137b-d234-4c2f-ad8a-eb48e972dcef\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") "
Sep 08 15:35:09.249665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.249523    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2242137b-d234-4c2f-ad8a-eb48e972dcef-openshift-service-ca\") pod \"2242137b-d234-4c2f-ad8a-eb48e972dcef\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") "
Sep 08 15:35:09.249665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.249563    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc5s9\" (UniqueName: \"kubernetes.io/projected/2242137b-d234-4c2f-ad8a-eb48e972dcef-kube-api-access-kc5s9\") pod \"2242137b-d234-4c2f-ad8a-eb48e972dcef\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") "
Sep 08 15:35:09.249665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.249660    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-metrics-certs\") pod \"2242137b-d234-4c2f-ad8a-eb48e972dcef\" (UID: \"2242137b-d234-4c2f-ad8a-eb48e972dcef\") "
Sep 08 15:35:09.250006 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.249978    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2242137b-d234-4c2f-ad8a-eb48e972dcef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2242137b-d234-4c2f-ad8a-eb48e972dcef" (UID: "2242137b-d234-4c2f-ad8a-eb48e972dcef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:09.252016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.251987    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2242137b-d234-4c2f-ad8a-eb48e972dcef" (UID: "2242137b-d234-4c2f-ad8a-eb48e972dcef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:35:09.252200 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.252178    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2242137b-d234-4c2f-ad8a-eb48e972dcef" (UID: "2242137b-d234-4c2f-ad8a-eb48e972dcef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:35:09.255543 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.255509    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2242137b-d234-4c2f-ad8a-eb48e972dcef-kube-api-access-kc5s9" (OuterVolumeSpecName: "kube-api-access-kc5s9") pod "2242137b-d234-4c2f-ad8a-eb48e972dcef" (UID: "2242137b-d234-4c2f-ad8a-eb48e972dcef"). InnerVolumeSpecName "kube-api-access-kc5s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:09.286624 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.286606    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:35:09.291352 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.291327    2575 generic.go:356] "Generic (PLEG): container finished" podID="738e6dda-9a86-4dae-a495-a2880582cc75" containerID="ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99" exitCode=0
Sep 08 15:35:09.291441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.291369    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"
Sep 08 15:35:09.291441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.291408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw" event={"ID":"738e6dda-9a86-4dae-a495-a2880582cc75","Type":"ContainerDied","Data":"ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99"}
Sep 08 15:35:09.291533 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.291455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw" event={"ID":"738e6dda-9a86-4dae-a495-a2880582cc75","Type":"ContainerDied","Data":"3b388ab485977958293df6f2e1e8a2b6fe52ff2be0552b14326ebf3fd83be49f"}
Sep 08 15:35:09.291533 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.291483    2575 scope.go:117] "RemoveContainer" containerID="ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99"
Sep 08 15:35:09.292707 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.292685    2575 generic.go:356] "Generic (PLEG): container finished" podID="2242137b-d234-4c2f-ad8a-eb48e972dcef" containerID="59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3" exitCode=0
Sep 08 15:35:09.292804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.292739    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"
Sep 08 15:35:09.292804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.292770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" event={"ID":"2242137b-d234-4c2f-ad8a-eb48e972dcef","Type":"ContainerDied","Data":"59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3"}
Sep 08 15:35:09.292937 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.292819    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94" event={"ID":"2242137b-d234-4c2f-ad8a-eb48e972dcef","Type":"ContainerDied","Data":"2e445bea0c44783ff07e15e9cd87dca0ec3ee7ce48c580f5a933cca76a0dd9a0"}
Sep 08 15:35:09.301193 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.301176    2575 scope.go:117] "RemoveContainer" containerID="ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99"
Sep 08 15:35:09.301550 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:35:09.301529    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99\": container with ID starting with ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99 not found: ID does not exist" containerID="ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99"
Sep 08 15:35:09.301640 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.301561    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99"} err="failed to get container status \"ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99\": rpc error: code = NotFound desc = could not find container \"ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99\": container with ID starting with ab898f4722d1ae8f10b7d8411e46e7d61c377631660fbe7e802bea8563457e99 not found: ID does not exist"
Sep 08 15:35:09.301640 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.301587    2575 scope.go:117] "RemoveContainer" containerID="59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3"
Sep 08 15:35:09.313148 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.313119    2575 scope.go:117] "RemoveContainer" containerID="59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3"
Sep 08 15:35:09.313463 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:35:09.313442    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3\": container with ID starting with 59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3 not found: ID does not exist" containerID="59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3"
Sep 08 15:35:09.313537 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.313473    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3"} err="failed to get container status \"59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3\": rpc error: code = NotFound desc = could not find container \"59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3\": container with ID starting with 59fd4546ae05c650dfa7cef9ec4af5313dd7dbaa1eb2e12a65637b98515c2be3 not found: ID does not exist"
Sep 08 15:35:09.331059 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.331035    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"]
Sep 08 15:35:09.336558 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.336537    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e39762-7574978f5f-l7q94"]
Sep 08 15:35:09.350685 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.350666    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738e6dda-9a86-4dae-a495-a2880582cc75-plugins-config-volume\") pod \"738e6dda-9a86-4dae-a495-a2880582cc75\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") "
Sep 08 15:35:09.350773 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.350732    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5jg6\" (UniqueName: \"kubernetes.io/projected/738e6dda-9a86-4dae-a495-a2880582cc75-kube-api-access-x5jg6\") pod \"738e6dda-9a86-4dae-a495-a2880582cc75\" (UID: \"738e6dda-9a86-4dae-a495-a2880582cc75\") "
Sep 08 15:35:09.350969 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.350954    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:09.351031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.350976    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2242137b-d234-4c2f-ad8a-eb48e972dcef-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:09.351031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.350989    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2242137b-d234-4c2f-ad8a-eb48e972dcef-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:09.351031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.351002    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5s9\" (UniqueName: \"kubernetes.io/projected/2242137b-d234-4c2f-ad8a-eb48e972dcef-kube-api-access-kc5s9\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:09.351162 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.351097    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738e6dda-9a86-4dae-a495-a2880582cc75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "738e6dda-9a86-4dae-a495-a2880582cc75" (UID: "738e6dda-9a86-4dae-a495-a2880582cc75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:09.352765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.352745    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738e6dda-9a86-4dae-a495-a2880582cc75-kube-api-access-x5jg6" (OuterVolumeSpecName: "kube-api-access-x5jg6") pod "738e6dda-9a86-4dae-a495-a2880582cc75" (UID: "738e6dda-9a86-4dae-a495-a2880582cc75"). InnerVolumeSpecName "kube-api-access-x5jg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:09.441114 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.441091    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2242137b-d234-4c2f-ad8a-eb48e972dcef" path="/var/lib/kubelet/pods/2242137b-d234-4c2f-ad8a-eb48e972dcef/volumes"
Sep 08 15:35:09.451554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.451511    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738e6dda-9a86-4dae-a495-a2880582cc75-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:09.451554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.451529    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5jg6\" (UniqueName: \"kubernetes.io/projected/738e6dda-9a86-4dae-a495-a2880582cc75-kube-api-access-x5jg6\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:09.607694 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.607672    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"]
Sep 08 15:35:09.612031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:09.612009    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e39762-6bd48d4dc9-ldpjw"]
Sep 08 15:35:11.108572 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:11.108537    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:35:11.440449 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:11.440422    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738e6dda-9a86-4dae-a495-a2880582cc75" path="/var/lib/kubelet/pods/738e6dda-9a86-4dae-a495-a2880582cc75/volumes"
Sep 08 15:35:17.105646 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.105613    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"]
Sep 08 15:35:17.106087 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106071    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy"
Sep 08 15:35:17.106135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106090    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy"
Sep 08 15:35:17.106135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106112    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b8156b-a8b8-414e-afec-8996aebbf706" containerName="main"
Sep 08 15:35:17.106135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106121    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b8156b-a8b8-414e-afec-8996aebbf706" containerName="main"
Sep 08 15:35:17.106135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106133    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2242137b-d234-4c2f-ad8a-eb48e972dcef" containerName="maas-api"
Sep 08 15:35:17.106254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106142    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2242137b-d234-4c2f-ad8a-eb48e972dcef" containerName="maas-api"
Sep 08 15:35:17.106254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106156    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7de9878-6f56-4475-a72b-7e2b4508d95e" containerName="main"
Sep 08 15:35:17.106254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106163    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7de9878-6f56-4475-a72b-7e2b4508d95e" containerName="main"
Sep 08 15:35:17.106254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106179    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738e6dda-9a86-4dae-a495-a2880582cc75" containerName="payload-processing"
Sep 08 15:35:17.106254 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106187    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="738e6dda-9a86-4dae-a495-a2880582cc75" containerName="payload-processing"
Sep 08 15:35:17.106401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106254    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy"
Sep 08 15:35:17.106401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106267    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7de9878-6f56-4475-a72b-7e2b4508d95e" containerName="main"
Sep 08 15:35:17.106401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106276    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="738e6dda-9a86-4dae-a495-a2880582cc75" containerName="payload-processing"
Sep 08 15:35:17.106401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106285    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy"
Sep 08 15:35:17.106401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106293    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2242137b-d234-4c2f-ad8a-eb48e972dcef" containerName="maas-api"
Sep 08 15:35:17.106401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.106300    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b8156b-a8b8-414e-afec-8996aebbf706" containerName="main"
Sep 08 15:35:17.111259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.111242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.116695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.116672    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9ca1fc-openshift-default-dockercfg-v59gw\""
Sep 08 15:35:17.128886 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.128862    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"]
Sep 08 15:35:17.201882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.201836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-socket\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.201910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-podinfo\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.201938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-token\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.201957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-envoy\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.201992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhzr\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-kube-api-access-9nhzr\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.202024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-data\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202061 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.202061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-certs\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.202115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c443c10-40ed-4406-9c12-f550fd02bad7-istiod-ca-cert\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.202371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.202153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-credential-socket\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303517 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-certs\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303517 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c443c10-40ed-4406-9c12-f550fd02bad7-istiod-ca-cert\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-credential-socket\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-socket\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-podinfo\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-token\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-envoy\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.303744 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhzr\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-kube-api-access-9nhzr\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.304083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-data\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.304083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.303965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-certs\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.304083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.304035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-credential-socket\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.304242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.304090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-data\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.304302 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.304273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-socket\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.304356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.304328    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c443c10-40ed-4406-9c12-f550fd02bad7-istiod-ca-cert\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.306728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.306699    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-envoy\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.306880 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.306861    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-podinfo\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.318610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.318588    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-token\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.330620 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.330590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhzr\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-kube-api-access-9nhzr\") pod \"e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") " pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.421193 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.421158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:17.707743 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.707709    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"]
Sep 08 15:35:17.708566 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:35:17.708544    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c443c10_40ed_4406_9c12_f550fd02bad7.slice/crio-0228a40c858487cfb7eb3de6a8e238b330c502b97085c879ed9274314f992afc WatchSource:0}: Error finding container 0228a40c858487cfb7eb3de6a8e238b330c502b97085c879ed9274314f992afc: Status 404 returned error can't find the container with id 0228a40c858487cfb7eb3de6a8e238b330c502b97085c879ed9274314f992afc
Sep 08 15:35:17.711060 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.711029    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:35:17.711133 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.711104    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:35:17.711171 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:17.711136    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:35:18.347018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.346981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" event={"ID":"7c443c10-40ed-4406-9c12-f550fd02bad7","Type":"ContainerStarted","Data":"d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"}
Sep 08 15:35:18.347567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.347045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" event={"ID":"7c443c10-40ed-4406-9c12-f550fd02bad7","Type":"ContainerStarted","Data":"0228a40c858487cfb7eb3de6a8e238b330c502b97085c879ed9274314f992afc"}
Sep 08 15:35:18.373020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.372973    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podStartSLOduration=1.372956204 podStartE2EDuration="1.372956204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:18.371811026 +0000 UTC m=+1067.447103319" watchObservedRunningTime="2026-09-08 15:35:18.372956204 +0000 UTC m=+1067.448248569"
Sep 08 15:35:18.422353 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.422328    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:18.423677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.423656    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 08 15:35:18.423789 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.423704    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 08 15:35:18.974044 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.974015    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"]
Sep 08 15:35:18.974239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.974224    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r" podUID="6055c843-2195-4409-ab3d-f38fd63fc281" containerName="payload-processing" containerID="cri-o://b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9" gracePeriod=30
Sep 08 15:35:18.981858 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.981820    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"]
Sep 08 15:35:18.982030 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:18.982010    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc" podUID="1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" containerName="payload-pre-processing" containerID="cri-o://f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390" gracePeriod=30
Sep 08 15:35:19.242856 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.242821    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:35:19.245579 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.245559    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:35:19.322733 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.322703    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x88z4\" (UniqueName: \"kubernetes.io/projected/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-kube-api-access-x88z4\") pod \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") "
Sep 08 15:35:19.322883 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.322747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6055c843-2195-4409-ab3d-f38fd63fc281-plugins-config-volume\") pod \"6055c843-2195-4409-ab3d-f38fd63fc281\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") "
Sep 08 15:35:19.322883 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.322871    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5pwc\" (UniqueName: \"kubernetes.io/projected/6055c843-2195-4409-ab3d-f38fd63fc281-kube-api-access-b5pwc\") pod \"6055c843-2195-4409-ab3d-f38fd63fc281\" (UID: \"6055c843-2195-4409-ab3d-f38fd63fc281\") "
Sep 08 15:35:19.323012 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.322931    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-plugins-config-volume\") pod \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\" (UID: \"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd\") "
Sep 08 15:35:19.323313 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.323281    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6055c843-2195-4409-ab3d-f38fd63fc281-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6055c843-2195-4409-ab3d-f38fd63fc281" (UID: "6055c843-2195-4409-ab3d-f38fd63fc281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:19.323434 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.323354    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" (UID: "1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:19.325080 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.325050    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-kube-api-access-x88z4" (OuterVolumeSpecName: "kube-api-access-x88z4") pod "1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" (UID: "1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd"). InnerVolumeSpecName "kube-api-access-x88z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:19.325400 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.325375    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6055c843-2195-4409-ab3d-f38fd63fc281-kube-api-access-b5pwc" (OuterVolumeSpecName: "kube-api-access-b5pwc") pod "6055c843-2195-4409-ab3d-f38fd63fc281" (UID: "6055c843-2195-4409-ab3d-f38fd63fc281"). InnerVolumeSpecName "kube-api-access-b5pwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:19.353713 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.353689    2575 generic.go:356] "Generic (PLEG): container finished" podID="6055c843-2195-4409-ab3d-f38fd63fc281" containerID="b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9" exitCode=0
Sep 08 15:35:19.354093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.353740    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"
Sep 08 15:35:19.354093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.353772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r" event={"ID":"6055c843-2195-4409-ab3d-f38fd63fc281","Type":"ContainerDied","Data":"b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9"}
Sep 08 15:35:19.354093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.353820    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r" event={"ID":"6055c843-2195-4409-ab3d-f38fd63fc281","Type":"ContainerDied","Data":"ffe3b420ec3b73e51fe869c05e87df0dc07b7ea70709440a03c8f4a4a2c64b64"}
Sep 08 15:35:19.354093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.353859    2575 scope.go:117] "RemoveContainer" containerID="b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9"
Sep 08 15:35:19.355339 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.355317    2575 generic.go:356] "Generic (PLEG): container finished" podID="1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" containerID="f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390" exitCode=0
Sep 08 15:35:19.355438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.355351    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"
Sep 08 15:35:19.355438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.355388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc" event={"ID":"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd","Type":"ContainerDied","Data":"f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390"}
Sep 08 15:35:19.355438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.355429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc" event={"ID":"1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd","Type":"ContainerDied","Data":"195f5e9c33507ce1608fe5167b8d5570e5be8d9f8ef14d83b2362aeb19f9b63a"}
Sep 08 15:35:19.363292 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.363278    2575 scope.go:117] "RemoveContainer" containerID="b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9"
Sep 08 15:35:19.363564 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:35:19.363541    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9\": container with ID starting with b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9 not found: ID does not exist" containerID="b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9"
Sep 08 15:35:19.363670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.363576    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9"} err="failed to get container status \"b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9\": rpc error: code = NotFound desc = could not find container \"b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9\": container with ID starting with b2275715d39d452f3b5ac9b6e423c6db21b073701b8464fc5aaaa301d227c7d9 not found: ID does not exist"
Sep 08 15:35:19.363670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.363599    2575 scope.go:117] "RemoveContainer" containerID="f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390"
Sep 08 15:35:19.371020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.370997    2575 scope.go:117] "RemoveContainer" containerID="f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390"
Sep 08 15:35:19.371254 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:35:19.371237    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390\": container with ID starting with f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390 not found: ID does not exist" containerID="f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390"
Sep 08 15:35:19.371308 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.371261    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390"} err="failed to get container status \"f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390\": rpc error: code = NotFound desc = could not find container \"f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390\": container with ID starting with f0fe3a4903e02612a50cd1867040f7fbd4b6fabe5f2ac523f00012dbb9991390 not found: ID does not exist"
Sep 08 15:35:19.403740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.403719    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"]
Sep 08 15:35:19.415859 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.415822    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9fbfa439-54b7d957fd-h4bvc"]
Sep 08 15:35:19.425086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.424596    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:19.425086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.424622    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x88z4\" (UniqueName: \"kubernetes.io/projected/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd-kube-api-access-x88z4\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:19.425086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.424637    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6055c843-2195-4409-ab3d-f38fd63fc281-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:19.425086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.424658    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5pwc\" (UniqueName: \"kubernetes.io/projected/6055c843-2195-4409-ab3d-f38fd63fc281-kube-api-access-b5pwc\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:35:19.430409 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.430388    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:19.441131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.441096    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" path="/var/lib/kubelet/pods/1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd/volumes"
Sep 08 15:35:19.441484 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.441459    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"]
Sep 08 15:35:19.443943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:19.443925    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9fbfa439-67856bdc45-2n77r"]
Sep 08 15:35:20.362829 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:20.362777    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:20.363827 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:20.363803    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:21.222674 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:21.222636    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:35:21.440934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:21.440896    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6055c843-2195-4409-ab3d-f38fd63fc281" path="/var/lib/kubelet/pods/6055c843-2195-4409-ab3d-f38fd63fc281/volumes"
Sep 08 15:35:22.648286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648216    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"]
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648515    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648525    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2116d54-3f72-4a7e-960d-7582e423136b" containerName="istio-proxy"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648532    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6055c843-2195-4409-ab3d-f38fd63fc281" containerName="payload-processing"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648538    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6055c843-2195-4409-ab3d-f38fd63fc281" containerName="payload-processing"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648553    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" containerName="payload-pre-processing"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648559    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" containerName="payload-pre-processing"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648608    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fba4bc7-37bc-40f4-8368-bf9d5a4f6ffd" containerName="payload-pre-processing"
Sep 08 15:35:22.648625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.648618    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6055c843-2195-4409-ab3d-f38fd63fc281" containerName="payload-processing"
Sep 08 15:35:22.652884 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.652867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.655680 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.655655    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-9ca1fc-dockercfg-fhc88\""
Sep 08 15:35:22.655870 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.655856    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9ca1fc\""
Sep 08 15:35:22.668783 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.668754    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"]
Sep 08 15:35:22.747371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.747342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwgmv\" (UniqueName: \"kubernetes.io/projected/8aea44b8-26dc-41bd-9981-ce793c931939-kube-api-access-rwgmv\") pod \"payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.747561 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.747406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aea44b8-26dc-41bd-9981-ce793c931939-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.848305 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.848279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aea44b8-26dc-41bd-9981-ce793c931939-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.848448 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.848333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgmv\" (UniqueName: \"kubernetes.io/projected/8aea44b8-26dc-41bd-9981-ce793c931939-kube-api-access-rwgmv\") pod \"payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.849001 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.848983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aea44b8-26dc-41bd-9981-ce793c931939-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.858358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.858326    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgmv\" (UniqueName: \"kubernetes.io/projected/8aea44b8-26dc-41bd-9981-ce793c931939-kube-api-access-rwgmv\") pod \"payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:22.962078 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:22.962022    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:23.235456 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:23.235430    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"]
Sep 08 15:35:23.237629 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:35:23.237601    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aea44b8_26dc_41bd_9981_ce793c931939.slice/crio-6b3e52a59d8040cbb0ed88f29772b3c5690bc3574c3de6a787bfea3cf56ba050 WatchSource:0}: Error finding container 6b3e52a59d8040cbb0ed88f29772b3c5690bc3574c3de6a787bfea3cf56ba050: Status 404 returned error can't find the container with id 6b3e52a59d8040cbb0ed88f29772b3c5690bc3574c3de6a787bfea3cf56ba050
Sep 08 15:35:23.380932 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:23.380896    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf" event={"ID":"8aea44b8-26dc-41bd-9981-ce793c931939","Type":"ContainerStarted","Data":"433ba5a84ba7b5e602d4bad1337e1cdf5c656337a438d09c94330c65e565a983"}
Sep 08 15:35:23.381094 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:23.380958    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf" event={"ID":"8aea44b8-26dc-41bd-9981-ce793c931939","Type":"ContainerStarted","Data":"6b3e52a59d8040cbb0ed88f29772b3c5690bc3574c3de6a787bfea3cf56ba050"}
Sep 08 15:35:23.381094 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:23.381008    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:23.401610 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:23.401564    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf" podStartSLOduration=1.40155071 podStartE2EDuration="1.40155071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:23.399594426 +0000 UTC m=+1072.474886737" watchObservedRunningTime="2026-09-08 15:35:23.40155071 +0000 UTC m=+1072.476843037"
Sep 08 15:35:30.314871 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:30.314824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:35:34.387979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:34.387946    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:35:36.222156 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.222124    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:53264->10.133.0.51:15021: read: connection reset by peer" start-of-body=
Sep 08 15:35:36.222545 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.222202    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:53264->10.133.0.51:15021: read: connection reset by peer"
Sep 08 15:35:36.439531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.439504    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerID="d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255" exitCode=0
Sep 08 15:35:36.439672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.439569    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" event={"ID":"7c443c10-40ed-4406-9c12-f550fd02bad7","Type":"ContainerDied","Data":"d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"}
Sep 08 15:35:36.439955 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.439940    2575 scope.go:117] "RemoveContainer" containerID="d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"
Sep 08 15:35:36.440979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.440947    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:35:36.441046 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.441023    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:35:36.441092 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:36.441058    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 08 15:35:37.445293 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:37.445259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" event={"ID":"7c443c10-40ed-4406-9c12-f550fd02bad7","Type":"ContainerStarted","Data":"2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2"}
Sep 08 15:35:38.422366 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:38.422331    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:38.422638 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:38.422616    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 08 15:35:38.422722 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:38.422703    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 08 15:35:39.422197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:39.422158    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 08 15:35:39.422651 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:39.422220    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 08 15:35:40.422171 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:40.422126    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 08 15:35:40.422383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:40.422192    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 08 15:35:41.422020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:41.421977    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 08 15:35:41.422238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:41.422043    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 08 15:35:42.422731 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:42.422692    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 08 15:35:42.423151 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:42.422761    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 08 15:35:43.578013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:43.577983    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:43.578473 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:43.578261    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:43.579013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:43.578991    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:35:46.090691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.090652    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"]
Sep 08 15:35:46.093526 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.093506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.097971 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.097953    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-03e8df-dockercfg-fkgbf\""
Sep 08 15:35:46.098514 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.098501    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-03e8df\""
Sep 08 15:35:46.107752 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.107732    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"]
Sep 08 15:35:46.249269 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.249238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.249436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.249287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwnr\" (UniqueName: \"kubernetes.io/projected/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-kube-api-access-hdwnr\") pod \"payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.350197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.350126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.350197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.350174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwnr\" (UniqueName: \"kubernetes.io/projected/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-kube-api-access-hdwnr\") pod \"payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.350779 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.350760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.360016 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.359993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwnr\" (UniqueName: \"kubernetes.io/projected/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-kube-api-access-hdwnr\") pod \"payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.403778 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.403754    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:46.674018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:46.673973    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"]
Sep 08 15:35:46.676228 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:35:46.676198    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f80f3ae_ad7a_43d0_98b8_f052f27690ad.slice/crio-ccd9f1be782223e541b9e15dd2be5db4596f35cc1a1b918a9e6707bf5bb8bba1 WatchSource:0}: Error finding container ccd9f1be782223e541b9e15dd2be5db4596f35cc1a1b918a9e6707bf5bb8bba1: Status 404 returned error can't find the container with id ccd9f1be782223e541b9e15dd2be5db4596f35cc1a1b918a9e6707bf5bb8bba1
Sep 08 15:35:47.492227 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:47.492186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r" event={"ID":"6f80f3ae-ad7a-43d0-98b8-f052f27690ad","Type":"ContainerStarted","Data":"eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf"}
Sep 08 15:35:47.492592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:47.492243    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r" event={"ID":"6f80f3ae-ad7a-43d0-98b8-f052f27690ad","Type":"ContainerStarted","Data":"ccd9f1be782223e541b9e15dd2be5db4596f35cc1a1b918a9e6707bf5bb8bba1"}
Sep 08 15:35:47.492592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:47.492293    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:47.531626 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:47.531578    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r" podStartSLOduration=1.531562465 podStartE2EDuration="1.531562465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:47.529653744 +0000 UTC m=+1096.604946037" watchObservedRunningTime="2026-09-08 15:35:47.531562465 +0000 UTC m=+1096.606854802"
Sep 08 15:35:50.443840 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.443797    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"]
Sep 08 15:35:50.446980 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.446950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.453805 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.453766    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:35:50.454026 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.454009    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-9fb31f-serving-cert\""
Sep 08 15:35:50.454108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.454036    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2xlj4\""
Sep 08 15:35:50.454108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.454087    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:35:50.455496 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.455474    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:35:50.483970 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.483940    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"]
Sep 08 15:35:50.585594 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.585566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8r88\" (UniqueName: \"kubernetes.io/projected/2985b6fb-34d3-4093-99a1-14fbd57d41af-kube-api-access-k8r88\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.585756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.585697    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.585756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.585728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.585900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.585762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985b6fb-34d3-4093-99a1-14fbd57d41af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.610144 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.610117    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:35:50.687156 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.687123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.687156 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.687156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.687386 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.687185    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985b6fb-34d3-4093-99a1-14fbd57d41af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.687386 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.687354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8r88\" (UniqueName: \"kubernetes.io/projected/2985b6fb-34d3-4093-99a1-14fbd57d41af-kube-api-access-k8r88\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.687905 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.687882    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985b6fb-34d3-4093-99a1-14fbd57d41af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.689911 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.689883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.689911 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.689903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.702940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.702872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8r88\" (UniqueName: \"kubernetes.io/projected/2985b6fb-34d3-4093-99a1-14fbd57d41af-kube-api-access-k8r88\") pod \"maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:50.758983 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:50.758957    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:51.036418 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.036390    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"]
Sep 08 15:35:51.038332 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:35:51.038311    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2985b6fb_34d3_4093_99a1_14fbd57d41af.slice/crio-0089160b56554c92f6c282814004ef5511af0954968599a05c3af9eb143af5ec WatchSource:0}: Error finding container 0089160b56554c92f6c282814004ef5511af0954968599a05c3af9eb143af5ec: Status 404 returned error can't find the container with id 0089160b56554c92f6c282814004ef5511af0954968599a05c3af9eb143af5ec
Sep 08 15:35:51.039627 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.039603    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:35:51.513588 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.513552    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d" event={"ID":"2985b6fb-34d3-4093-99a1-14fbd57d41af","Type":"ContainerStarted","Data":"0089160b56554c92f6c282814004ef5511af0954968599a05c3af9eb143af5ec"}
Sep 08 15:35:51.608592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.608567    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"]
Sep 08 15:35:51.611381 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.611362    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.615717 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.615695    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9fb31f-dockercfg-zvbs4\""
Sep 08 15:35:51.615835 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.615788    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9fb31f\""
Sep 08 15:35:51.635998 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.635655    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"]
Sep 08 15:35:51.696376 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.696349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52faf85-1168-4a5d-8b67-a49df1f373ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.696501 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.696384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szwk\" (UniqueName: \"kubernetes.io/projected/d52faf85-1168-4a5d-8b67-a49df1f373ce-kube-api-access-4szwk\") pod \"payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.797997 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.797909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52faf85-1168-4a5d-8b67-a49df1f373ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.797997 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.797985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szwk\" (UniqueName: \"kubernetes.io/projected/d52faf85-1168-4a5d-8b67-a49df1f373ce-kube-api-access-4szwk\") pod \"payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.798476 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.798458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52faf85-1168-4a5d-8b67-a49df1f373ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.810114 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.810086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szwk\" (UniqueName: \"kubernetes.io/projected/d52faf85-1168-4a5d-8b67-a49df1f373ce-kube-api-access-4szwk\") pod \"payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:51.922260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:51.922229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:52.195018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:52.194991    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"]
Sep 08 15:35:52.196703 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:35:52.196679    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52faf85_1168_4a5d_8b67_a49df1f373ce.slice/crio-fa22da4623c4da868f605e63a651e4e87f75295463b62231baaaffe56e9d4161 WatchSource:0}: Error finding container fa22da4623c4da868f605e63a651e4e87f75295463b62231baaaffe56e9d4161: Status 404 returned error can't find the container with id fa22da4623c4da868f605e63a651e4e87f75295463b62231baaaffe56e9d4161
Sep 08 15:35:52.521920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:52.521815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd" event={"ID":"d52faf85-1168-4a5d-8b67-a49df1f373ce","Type":"ContainerStarted","Data":"eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319"}
Sep 08 15:35:52.521920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:52.521884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd" event={"ID":"d52faf85-1168-4a5d-8b67-a49df1f373ce","Type":"ContainerStarted","Data":"fa22da4623c4da868f605e63a651e4e87f75295463b62231baaaffe56e9d4161"}
Sep 08 15:35:52.521920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:52.521906    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:35:52.548487 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:52.548441    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd" podStartSLOduration=1.548425983 podStartE2EDuration="1.548425983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:52.546968479 +0000 UTC m=+1101.622260797" watchObservedRunningTime="2026-09-08 15:35:52.548425983 +0000 UTC m=+1101.623718276"
Sep 08 15:35:53.528256 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:53.528228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d" event={"ID":"2985b6fb-34d3-4093-99a1-14fbd57d41af","Type":"ContainerStarted","Data":"83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd"}
Sep 08 15:35:53.528672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:53.528356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:53.549804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:53.549755    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d" podStartSLOduration=1.754223106 podStartE2EDuration="3.54974095s" totalImagesPullingTime="1.795517844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:53.548964501 +0000 UTC m=+1102.624256820" watchObservedRunningTime="2026-09-08 15:35:53.54974095 +0000 UTC m=+1102.625033243"
Sep 08 15:35:58.499546 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:58.499509    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:35:59.538378 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:59.538342    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:35:59.699416 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:35:59.699383    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:36:03.530450 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:03.530421    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:36:13.906733 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:13.906702    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"]
Sep 08 15:36:13.910472 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:13.910448    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:13.914306 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:13.914279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a7ae4e-serving-cert\""
Sep 08 15:36:13.926824 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:13.926783    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"]
Sep 08 15:36:14.073047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.073014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.073190 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.073072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.073245 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.073213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b63a8a-2ea3-4317-94bc-248015e992ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.073292 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.073262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78rs\" (UniqueName: \"kubernetes.io/projected/70b63a8a-2ea3-4317-94bc-248015e992ca-kube-api-access-v78rs\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.174474 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.174393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b63a8a-2ea3-4317-94bc-248015e992ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.174474 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.174441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78rs\" (UniqueName: \"kubernetes.io/projected/70b63a8a-2ea3-4317-94bc-248015e992ca-kube-api-access-v78rs\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.174658 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.174481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.174658 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.174531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.174769 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:36:14.174661    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-serving-cert: secret "maas-api-e2e-shared-b-w2-a7ae4e-serving-cert" not found
Sep 08 15:36:14.174769 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:36:14.174731    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls podName:70b63a8a-2ea3-4317-94bc-248015e992ca nodeName:}" failed. No retries permitted until 2026-09-08 15:36:14.674710795 +0000 UTC m=+1123.750003072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" (UID: "70b63a8a-2ea3-4317-94bc-248015e992ca") : secret "maas-api-e2e-shared-b-w2-a7ae4e-serving-cert" not found
Sep 08 15:36:14.175037 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.175018    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b63a8a-2ea3-4317-94bc-248015e992ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.177030 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.177002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.188660 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.188636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78rs\" (UniqueName: \"kubernetes.io/projected/70b63a8a-2ea3-4317-94bc-248015e992ca-kube-api-access-v78rs\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.678397 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.678363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.680792 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.680758    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.824238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.824211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:14.980713 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.980649    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"]
Sep 08 15:36:14.984488 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.984473    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:14.987461 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.987439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a7ae4e-dockercfg-qlhwm\""
Sep 08 15:36:14.987560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:14.987466    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a7ae4e\""
Sep 08 15:36:15.001597 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.001574    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"]
Sep 08 15:36:15.112452 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.112253    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"]
Sep 08 15:36:15.114637 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:36:15.114605    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b63a8a_2ea3_4317_94bc_248015e992ca.slice/crio-c50571582b3b6ba9894e75865eb086b24d6b811e2bedc4ffbad08d8820f110cb WatchSource:0}: Error finding container c50571582b3b6ba9894e75865eb086b24d6b811e2bedc4ffbad08d8820f110cb: Status 404 returned error can't find the container with id c50571582b3b6ba9894e75865eb086b24d6b811e2bedc4ffbad08d8820f110cb
Sep 08 15:36:15.182923 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.182891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkc6l\" (UniqueName: \"kubernetes.io/projected/e70c1d36-888f-44a6-971c-8512b9ad5b9e-kube-api-access-tkc6l\") pod \"payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.183047 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.182935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70c1d36-888f-44a6-971c-8512b9ad5b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.283748 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.283688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkc6l\" (UniqueName: \"kubernetes.io/projected/e70c1d36-888f-44a6-971c-8512b9ad5b9e-kube-api-access-tkc6l\") pod \"payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.283748 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.283730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70c1d36-888f-44a6-971c-8512b9ad5b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.284356 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.284338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70c1d36-888f-44a6-971c-8512b9ad5b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.297808 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.297778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkc6l\" (UniqueName: \"kubernetes.io/projected/e70c1d36-888f-44a6-971c-8512b9ad5b9e-kube-api-access-tkc6l\") pod \"payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.596138 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.596070    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:15.646130 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.646074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" event={"ID":"70b63a8a-2ea3-4317-94bc-248015e992ca","Type":"ContainerStarted","Data":"05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078"}
Sep 08 15:36:15.646246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.646160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" event={"ID":"70b63a8a-2ea3-4317-94bc-248015e992ca","Type":"ContainerStarted","Data":"c50571582b3b6ba9894e75865eb086b24d6b811e2bedc4ffbad08d8820f110cb"}
Sep 08 15:36:15.646246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.646227    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:15.690093 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.690027    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" podStartSLOduration=2.400657632 podStartE2EDuration="2.690009095s" totalImagesPullingTime="289.351463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:36:15.688715181 +0000 UTC m=+1124.764007501" watchObservedRunningTime="2026-09-08 15:36:15.690009095 +0000 UTC m=+1124.765301388"
Sep 08 15:36:15.871740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:15.871715    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"]
Sep 08 15:36:15.873682 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:36:15.873630    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70c1d36_888f_44a6_971c_8512b9ad5b9e.slice/crio-3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1 WatchSource:0}: Error finding container 3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1: Status 404 returned error can't find the container with id 3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1
Sep 08 15:36:16.652288 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:16.652256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7" event={"ID":"e70c1d36-888f-44a6-971c-8512b9ad5b9e","Type":"ContainerStarted","Data":"7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e"}
Sep 08 15:36:16.652706 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:16.652444    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:16.652706 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:16.652482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7" event={"ID":"e70c1d36-888f-44a6-971c-8512b9ad5b9e","Type":"ContainerStarted","Data":"3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1"}
Sep 08 15:36:16.671931 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:16.671884    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7" podStartSLOduration=2.6718719440000003 podStartE2EDuration="2.671871944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:36:16.671095824 +0000 UTC m=+1125.746388119" watchObservedRunningTime="2026-09-08 15:36:16.671871944 +0000 UTC m=+1125.747164238"
Sep 08 15:36:19.401160 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.401123    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:36:19.513670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.508717    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"]
Sep 08 15:36:19.513670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.513619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.516786 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.516754    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"default-dockercfg-b8r6b\""
Sep 08 15:36:19.518236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518214    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"kube-root-ca.crt\""
Sep 08 15:36:19.518236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518227    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"auth-test-model-9ca1fc-kserve-self-signed-certs\""
Sep 08 15:36:19.518399 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn5fm\" (UniqueName: \"kubernetes.io/projected/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-kube-api-access-vn5fm\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.518399 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518291    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"openshift-service-ca.crt\""
Sep 08 15:36:19.518399 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tmp-dir\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.518399 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-dshm\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.518692 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-home\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.518692 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-model-cache\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.518692 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.518552    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tls-certs\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.524509 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.524490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"]
Sep 08 15:36:19.619342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tmp-dir\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-dshm\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619524 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-home\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619524 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-model-cache\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619524 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tls-certs\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619524 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn5fm\" (UniqueName: \"kubernetes.io/projected/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-kube-api-access-vn5fm\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619735 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619716    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tmp-dir\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-home\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.619924 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.619827    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-model-cache\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.621653 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.621636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-dshm\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.621952 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.621936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tls-certs\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.631300 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.631278    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn5fm\" (UniqueName: \"kubernetes.io/projected/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-kube-api-access-vn5fm\") pod \"auth-test-model-9ca1fc-kserve-986d76799-lsrzl\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:19.823875 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:19.823767    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:20.108181 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:20.108155    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"]
Sep 08 15:36:20.110054 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:36:20.110026    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cf5cd7f_f8ad_4b6b_9814_5f343d8a6cda.slice/crio-d1efdb3d70c52d0562be04b3a78141f7b99b7234f3781398d1e32179ae7fa113 WatchSource:0}: Error finding container d1efdb3d70c52d0562be04b3a78141f7b99b7234f3781398d1e32179ae7fa113: Status 404 returned error can't find the container with id d1efdb3d70c52d0562be04b3a78141f7b99b7234f3781398d1e32179ae7fa113
Sep 08 15:36:20.674471 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:20.674436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl" event={"ID":"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda","Type":"ContainerStarted","Data":"a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df"}
Sep 08 15:36:20.675003 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:20.674481    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl" event={"ID":"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda","Type":"ContainerStarted","Data":"d1efdb3d70c52d0562be04b3a78141f7b99b7234f3781398d1e32179ae7fa113"}
Sep 08 15:36:20.695128 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:20.695050    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl" podStartSLOduration=1.695026323 podStartE2EDuration="1.695026323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:36:20.694539704 +0000 UTC m=+1129.769832023" watchObservedRunningTime="2026-09-08 15:36:20.695026323 +0000 UTC m=+1129.770318615"
Sep 08 15:36:21.658646 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:21.658619    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:36:27.659728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:27.659699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:36:29.823956 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:29.823916    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:29.824462 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:29.823984    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:29.836406 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:29.836382    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:30.741603 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:30.741574    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:36:31.512298 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:31.512264    2575 scope.go:117] "RemoveContainer" containerID="300babfd8dd10c34adece1a08e69744d690f08fdb8751ccc410648885d98a9cf"
Sep 08 15:36:33.140041 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:33.140000    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:36:40.727366 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.727329    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"]
Sep 08 15:36:40.730591 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.730572    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.735193 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.735167    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"kube-root-ca.crt\""
Sep 08 15:36:40.736157 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.736136    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"openshift-service-ca.crt\""
Sep 08 15:36:40.736532 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.736487    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"default-dockercfg-9tg9m\""
Sep 08 15:36:40.736678 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.736515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"auth-test-model-9fb31f-kserve-self-signed-certs\""
Sep 08 15:36:40.745626 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.745600    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"]
Sep 08 15:36:40.795670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.795648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxbv\" (UniqueName: \"kubernetes.io/projected/f96a211d-d7de-4bea-bdfe-586b204d562e-kube-api-access-7rxbv\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.795788 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.795677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-tmp-dir\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.795788 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.795698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-dshm\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.795788 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.795724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96a211d-d7de-4bea-bdfe-586b204d562e-tls-certs\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.795788 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.795771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-model-cache\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.795988 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.795805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-home\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897084 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897051    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-home\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxbv\" (UniqueName: \"kubernetes.io/projected/f96a211d-d7de-4bea-bdfe-586b204d562e-kube-api-access-7rxbv\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-tmp-dir\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-dshm\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897400 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96a211d-d7de-4bea-bdfe-586b204d562e-tls-certs\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897400 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-model-cache\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897509 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897443    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-home\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897562 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-tmp-dir\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.897632 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.897615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-model-cache\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.899608 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.899582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-dshm\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.899738 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.899723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96a211d-d7de-4bea-bdfe-586b204d562e-tls-certs\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:40.908575 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:40.908545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxbv\" (UniqueName: \"kubernetes.io/projected/f96a211d-d7de-4bea-bdfe-586b204d562e-kube-api-access-7rxbv\") pod \"auth-test-model-9fb31f-kserve-566cb865b7-lvp9h\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:41.040874 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:41.040766    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:41.317400 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:41.317375    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"]
Sep 08 15:36:41.319311 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:36:41.319236    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf96a211d_d7de_4bea_bdfe_586b204d562e.slice/crio-e91994c555f25f59cb987a0df1302c3d9a590b6e86d255d23495c9bf1ee13a4f WatchSource:0}: Error finding container e91994c555f25f59cb987a0df1302c3d9a590b6e86d255d23495c9bf1ee13a4f: Status 404 returned error can't find the container with id e91994c555f25f59cb987a0df1302c3d9a590b6e86d255d23495c9bf1ee13a4f
Sep 08 15:36:41.785532 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:41.785489    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h" event={"ID":"f96a211d-d7de-4bea-bdfe-586b204d562e","Type":"ContainerStarted","Data":"7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53"}
Sep 08 15:36:41.785980 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:41.785550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h" event={"ID":"f96a211d-d7de-4bea-bdfe-586b204d562e","Type":"ContainerStarted","Data":"e91994c555f25f59cb987a0df1302c3d9a590b6e86d255d23495c9bf1ee13a4f"}
Sep 08 15:36:41.809086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:41.809042    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h" podStartSLOduration=1.809028338 podStartE2EDuration="1.809028338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:36:41.80817438 +0000 UTC m=+1150.883466690" watchObservedRunningTime="2026-09-08 15:36:41.809028338 +0000 UTC m=+1150.884320631"
Sep 08 15:36:44.399508 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:44.399471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:36:51.041540 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:51.041504    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:51.041540 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:51.041550    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:51.054167 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:51.054145    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:36:51.847960 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:36:51.847929    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:37:05.507176 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:05.507097    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:37:22.011070 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:22.011031    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:37:34.516127 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:34.516093    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:37:43.852816 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.852780    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"]
Sep 08 15:37:43.856052 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.856033    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.865953 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.865932    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4e499427\"/\"kube-root-ca.crt\""
Sep 08 15:37:43.866042 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.865962    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4e499427\"/\"openshift-service-ca.crt\""
Sep 08 15:37:43.866042 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.865934    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4e499427\"/\"default-dockercfg-xp5mj\""
Sep 08 15:37:43.866042 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.865934    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4e499427\"/\"e2e-lifecycle-model-4e499427-kserve-self-signed-certs\""
Sep 08 15:37:43.891457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.891432    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"]
Sep 08 15:37:43.895309 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.895282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-home\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.895430 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.895341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tmp-dir\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.895430 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.895402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-model-cache\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.895523 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.895434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzjl\" (UniqueName: \"kubernetes.io/projected/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-kube-api-access-dvzjl\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.895523 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.895482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tls-certs\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.895523 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.895506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-dshm\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.996531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tls-certs\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.996646 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-dshm\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.996646 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-home\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.996646 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tmp-dir\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.996799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-model-cache\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.996799 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzjl\" (UniqueName: \"kubernetes.io/projected/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-kube-api-access-dvzjl\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.997015 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.996994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tmp-dir\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.997076 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.997029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-home\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.997183 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.997167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-model-cache\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.999030 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.999002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-dshm\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:43.999217 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:43.999198    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tls-certs\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:44.012999 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:44.012978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzjl\" (UniqueName: \"kubernetes.io/projected/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-kube-api-access-dvzjl\") pod \"e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") " pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:44.164863 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:44.164807    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:44.446545 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:44.446516    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"]
Sep 08 15:37:44.448385 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:37:44.448340    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1ddef08_dbcd_4ea9_8361_6cbd1e8f9ddd.slice/crio-67f9317565d026a4472442f51a6e9f913ad3914926f288273fe853bff5b4e42b WatchSource:0}: Error finding container 67f9317565d026a4472442f51a6e9f913ad3914926f288273fe853bff5b4e42b: Status 404 returned error can't find the container with id 67f9317565d026a4472442f51a6e9f913ad3914926f288273fe853bff5b4e42b
Sep 08 15:37:45.124057 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:45.124014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf" event={"ID":"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd","Type":"ContainerStarted","Data":"06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422"}
Sep 08 15:37:45.124442 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:45.124066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf" event={"ID":"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd","Type":"ContainerStarted","Data":"67f9317565d026a4472442f51a6e9f913ad3914926f288273fe853bff5b4e42b"}
Sep 08 15:37:45.148434 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:45.148382    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf" podStartSLOduration=2.148367059 podStartE2EDuration="2.148367059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:45.147464148 +0000 UTC m=+1214.222756457" watchObservedRunningTime="2026-09-08 15:37:45.148367059 +0000 UTC m=+1214.223659346"
Sep 08 15:37:52.562222 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:52.562191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:37:54.165771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:54.165722    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:54.165771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:54.165784    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:54.178532 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:54.178506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:37:54.190189 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:37:54.190167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:38:15.131275 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:15.131235    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:38:29.024488 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:29.024407    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:38:36.333446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:36.333403    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"]
Sep 08 15:38:46.616279 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:46.616242    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:38:51.357290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.357236    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf" podUID="b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" containerName="main" containerID="cri-o://06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422" gracePeriod=45
Sep 08 15:38:51.600649 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.600622    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:38:51.754255 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754229    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-model-cache\") pod \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") "
Sep 08 15:38:51.754446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754277    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvzjl\" (UniqueName: \"kubernetes.io/projected/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-kube-api-access-dvzjl\") pod \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") "
Sep 08 15:38:51.754446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754310    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-home\") pod \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") "
Sep 08 15:38:51.754446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754344    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tls-certs\") pod \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") "
Sep 08 15:38:51.754446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754367    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tmp-dir\") pod \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") "
Sep 08 15:38:51.754446 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754386    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-dshm\") pod \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\" (UID: \"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd\") "
Sep 08 15:38:51.754700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754543    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" (UID: "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:38:51.754700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754614    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-home" (OuterVolumeSpecName: "home") pod "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" (UID: "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:38:51.754700 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.754628    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" (UID: "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:38:51.756696 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.756671    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-dshm" (OuterVolumeSpecName: "dshm") pod "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" (UID: "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:38:51.756815 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.756715    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-kube-api-access-dvzjl" (OuterVolumeSpecName: "kube-api-access-dvzjl") pod "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" (UID: "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd"). InnerVolumeSpecName "kube-api-access-dvzjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:38:51.756815 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.756731    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" (UID: "b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:38:51.855656 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.855634    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:38:51.855656 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.855655    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzjl\" (UniqueName: \"kubernetes.io/projected/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-kube-api-access-dvzjl\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:38:51.855782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.855665    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:38:51.855782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.855673    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:38:51.855782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.855682    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:38:51.855782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:51.855689    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:38:52.464882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.464254    2575 generic.go:356] "Generic (PLEG): container finished" podID="b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" containerID="06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422" exitCode=0
Sep 08 15:38:52.464882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.464380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf" event={"ID":"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd","Type":"ContainerDied","Data":"06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422"}
Sep 08 15:38:52.464882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.464418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf" event={"ID":"b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd","Type":"ContainerDied","Data":"67f9317565d026a4472442f51a6e9f913ad3914926f288273fe853bff5b4e42b"}
Sep 08 15:38:52.464882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.464441    2575 scope.go:117] "RemoveContainer" containerID="06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422"
Sep 08 15:38:52.464882 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.464620    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"
Sep 08 15:38:52.479838 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.479815    2575 scope.go:117] "RemoveContainer" containerID="06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422"
Sep 08 15:38:52.480242 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:38:52.480210    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422\": container with ID starting with 06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422 not found: ID does not exist" containerID="06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422"
Sep 08 15:38:52.480311 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.480256    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422"} err="failed to get container status \"06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422\": rpc error: code = NotFound desc = could not find container \"06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422\": container with ID starting with 06e8ff548f35433355134c36748ec50c471e5b3f00c0c728ab4d5a9c50e11422 not found: ID does not exist"
Sep 08 15:38:52.493022 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.492990    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"]
Sep 08 15:38:52.500931 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:52.500910    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4e499427/e2e-lifecycle-model-4e499427-kserve-55b89f48d7-f84mf"]
Sep 08 15:38:53.442442 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:38:53.442409    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" path="/var/lib/kubelet/pods/b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd/volumes"
Sep 08 15:39:03.111099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:03.111057    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:39:16.849319 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:16.849281    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"]
Sep 08 15:39:17.797583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:17.797551    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"]
Sep 08 15:39:20.900513 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:20.900483    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:39:31.868902 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:31.868818    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl" podUID="5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" containerName="main" containerID="cri-o://a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df" gracePeriod=45
Sep 08 15:39:32.117875 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.117829    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:39:32.176976 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.176949    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tls-certs\") pod \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") "
Sep 08 15:39:32.177106 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.176997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn5fm\" (UniqueName: \"kubernetes.io/projected/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-kube-api-access-vn5fm\") pod \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") "
Sep 08 15:39:32.177106 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tmp-dir\") pod \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") "
Sep 08 15:39:32.177229 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177144    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-model-cache\") pod \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") "
Sep 08 15:39:32.177293 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177247    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-dshm\") pod \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") "
Sep 08 15:39:32.177293 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177247    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" (UID: "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:32.177396 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177291    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-home\") pod \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\" (UID: \"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda\") "
Sep 08 15:39:32.177449 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177379    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" (UID: "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:32.177595 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177556    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-home" (OuterVolumeSpecName: "home") pod "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" (UID: "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:32.177680 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177668    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:32.177735 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177687    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:32.177735 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.177704    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:32.179373 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.179350    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-dshm" (OuterVolumeSpecName: "dshm") pod "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" (UID: "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:32.179662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.179642    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" (UID: "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:39:32.179757 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.179677    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-kube-api-access-vn5fm" (OuterVolumeSpecName: "kube-api-access-vn5fm") pod "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" (UID: "5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda"). InnerVolumeSpecName "kube-api-access-vn5fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:39:32.278500 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.278475    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:32.278500 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.278496    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn5fm\" (UniqueName: \"kubernetes.io/projected/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-kube-api-access-vn5fm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:32.278625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.278505    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:32.657635 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.657602    2575 generic.go:356] "Generic (PLEG): container finished" podID="5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" containerID="a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df" exitCode=0
Sep 08 15:39:32.657786 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.657652    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"
Sep 08 15:39:32.657786 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.657674    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl" event={"ID":"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda","Type":"ContainerDied","Data":"a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df"}
Sep 08 15:39:32.657786 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.657716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl" event={"ID":"5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda","Type":"ContainerDied","Data":"d1efdb3d70c52d0562be04b3a78141f7b99b7234f3781398d1e32179ae7fa113"}
Sep 08 15:39:32.657786 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.657736    2575 scope.go:117] "RemoveContainer" containerID="a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df"
Sep 08 15:39:32.667712 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.667694    2575 scope.go:117] "RemoveContainer" containerID="a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df"
Sep 08 15:39:32.668069 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:39:32.668049    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df\": container with ID starting with a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df not found: ID does not exist" containerID="a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df"
Sep 08 15:39:32.668135 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.668079    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df"} err="failed to get container status \"a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df\": rpc error: code = NotFound desc = could not find container \"a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df\": container with ID starting with a5667959cd3f929e9e95e5cbbb5a76938e8f38cc9144b2e9bafcc262e98bc9df not found: ID does not exist"
Sep 08 15:39:32.681962 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.681935    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"]
Sep 08 15:39:32.690618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.690588    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/auth-test-model-9ca1fc-kserve-986d76799-lsrzl"]
Sep 08 15:39:32.818655 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:32.818627    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h" podUID="f96a211d-d7de-4bea-bdfe-586b204d562e" containerName="main" containerID="cri-o://7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53" gracePeriod=45
Sep 08 15:39:33.062107 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.062087    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:39:33.187723 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.187660    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-dshm\") pod \"f96a211d-d7de-4bea-bdfe-586b204d562e\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") "
Sep 08 15:39:33.187914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.187757    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96a211d-d7de-4bea-bdfe-586b204d562e-tls-certs\") pod \"f96a211d-d7de-4bea-bdfe-586b204d562e\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") "
Sep 08 15:39:33.187914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.187775    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-tmp-dir\") pod \"f96a211d-d7de-4bea-bdfe-586b204d562e\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") "
Sep 08 15:39:33.187914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.187795    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rxbv\" (UniqueName: \"kubernetes.io/projected/f96a211d-d7de-4bea-bdfe-586b204d562e-kube-api-access-7rxbv\") pod \"f96a211d-d7de-4bea-bdfe-586b204d562e\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") "
Sep 08 15:39:33.187914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.187827    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-model-cache\") pod \"f96a211d-d7de-4bea-bdfe-586b204d562e\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") "
Sep 08 15:39:33.187914 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.187872    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-home\") pod \"f96a211d-d7de-4bea-bdfe-586b204d562e\" (UID: \"f96a211d-d7de-4bea-bdfe-586b204d562e\") "
Sep 08 15:39:33.188194 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.188161    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f96a211d-d7de-4bea-bdfe-586b204d562e" (UID: "f96a211d-d7de-4bea-bdfe-586b204d562e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:33.188194 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.188133    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f96a211d-d7de-4bea-bdfe-586b204d562e" (UID: "f96a211d-d7de-4bea-bdfe-586b204d562e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:33.188298 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.188210    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-home" (OuterVolumeSpecName: "home") pod "f96a211d-d7de-4bea-bdfe-586b204d562e" (UID: "f96a211d-d7de-4bea-bdfe-586b204d562e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:33.190145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.190121    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-dshm" (OuterVolumeSpecName: "dshm") pod "f96a211d-d7de-4bea-bdfe-586b204d562e" (UID: "f96a211d-d7de-4bea-bdfe-586b204d562e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:39:33.190247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.190145    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96a211d-d7de-4bea-bdfe-586b204d562e-kube-api-access-7rxbv" (OuterVolumeSpecName: "kube-api-access-7rxbv") pod "f96a211d-d7de-4bea-bdfe-586b204d562e" (UID: "f96a211d-d7de-4bea-bdfe-586b204d562e"). InnerVolumeSpecName "kube-api-access-7rxbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:39:33.190247 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.190179    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96a211d-d7de-4bea-bdfe-586b204d562e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f96a211d-d7de-4bea-bdfe-586b204d562e" (UID: "f96a211d-d7de-4bea-bdfe-586b204d562e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:39:33.288460 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.288437    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96a211d-d7de-4bea-bdfe-586b204d562e-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:33.288460 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.288457    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:33.288582 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.288466    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxbv\" (UniqueName: \"kubernetes.io/projected/f96a211d-d7de-4bea-bdfe-586b204d562e-kube-api-access-7rxbv\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:33.288582 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.288474    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:33.288582 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.288482    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:33.288582 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.288489    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96a211d-d7de-4bea-bdfe-586b204d562e-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:39:33.441713 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.441653    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" path="/var/lib/kubelet/pods/5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda/volumes"
Sep 08 15:39:33.665621 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.665591    2575 generic.go:356] "Generic (PLEG): container finished" podID="f96a211d-d7de-4bea-bdfe-586b204d562e" containerID="7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53" exitCode=0
Sep 08 15:39:33.665725 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.665633    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"
Sep 08 15:39:33.665725 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.665677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h" event={"ID":"f96a211d-d7de-4bea-bdfe-586b204d562e","Type":"ContainerDied","Data":"7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53"}
Sep 08 15:39:33.665804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.665725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h" event={"ID":"f96a211d-d7de-4bea-bdfe-586b204d562e","Type":"ContainerDied","Data":"e91994c555f25f59cb987a0df1302c3d9a590b6e86d255d23495c9bf1ee13a4f"}
Sep 08 15:39:33.665804 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.665746    2575 scope.go:117] "RemoveContainer" containerID="7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53"
Sep 08 15:39:33.674395 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.674374    2575 scope.go:117] "RemoveContainer" containerID="7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53"
Sep 08 15:39:33.674689 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:39:33.674668    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53\": container with ID starting with 7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53 not found: ID does not exist" containerID="7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53"
Sep 08 15:39:33.674734 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.674697    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53"} err="failed to get container status \"7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53\": rpc error: code = NotFound desc = could not find container \"7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53\": container with ID starting with 7d7baca458f1b13e9fa128f44c38cbbcccbdc988eeca7a42d17b774999700b53 not found: ID does not exist"
Sep 08 15:39:33.698045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.697992    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"]
Sep 08 15:39:33.705983 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:33.705960    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/auth-test-model-9fb31f-kserve-566cb865b7-lvp9h"]
Sep 08 15:39:34.133836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.133808    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"]
Sep 08 15:39:34.134271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134197    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" containerName="main"
Sep 08 15:39:34.134271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134208    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" containerName="main"
Sep 08 15:39:34.134271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134219    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96a211d-d7de-4bea-bdfe-586b204d562e" containerName="main"
Sep 08 15:39:34.134271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134225    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96a211d-d7de-4bea-bdfe-586b204d562e" containerName="main"
Sep 08 15:39:34.134271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134231    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" containerName="main"
Sep 08 15:39:34.134271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134237    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" containerName="main"
Sep 08 15:39:34.134463 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134299    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf5cd7f-f8ad-4b6b-9814-5f343d8a6cda" containerName="main"
Sep 08 15:39:34.134463 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134309    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96a211d-d7de-4bea-bdfe-586b204d562e" containerName="main"
Sep 08 15:39:34.134463 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.134316    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ddef08-dbcd-4ea9-8361-6cbd1e8f9ddd" containerName="main"
Sep 08 15:39:34.138478 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.138458    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.141836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.141811    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"kube-root-ca.crt\""
Sep 08 15:39:34.143243 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.143223    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"default-dockercfg-b8r6b\""
Sep 08 15:39:34.143243 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.143234    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"openshift-service-ca.crt\""
Sep 08 15:39:34.143417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.143250    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ca1fc\"/\"sub-test-model-9ca1fc-kserve-self-signed-certs\""
Sep 08 15:39:34.158977 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.158952    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"]
Sep 08 15:39:34.196257 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.196226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56971b2-107d-47bd-807c-208c709b2293-tls-certs\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.196367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.196264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-home\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.196367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.196278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-tmp-dir\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.196367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.196306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-model-cache\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.196473 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.196364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-dshm\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.196473 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.196392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4drl\" (UniqueName: \"kubernetes.io/projected/c56971b2-107d-47bd-807c-208c709b2293-kube-api-access-g4drl\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.297670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.297646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56971b2-107d-47bd-807c-208c709b2293-tls-certs\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.297780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.297675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-home\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.297780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.297691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-tmp-dir\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.297780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.297708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-model-cache\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.297780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.297743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-dshm\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.298005 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.297800    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4drl\" (UniqueName: \"kubernetes.io/projected/c56971b2-107d-47bd-807c-208c709b2293-kube-api-access-g4drl\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.298115 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.298093    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-home\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.298184 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.298139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-tmp-dir\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.298238 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.298191    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-model-cache\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.300145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.300127    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-dshm\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.300281 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.300264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56971b2-107d-47bd-807c-208c709b2293-tls-certs\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.307439 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.307417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4drl\" (UniqueName: \"kubernetes.io/projected/c56971b2-107d-47bd-807c-208c709b2293-kube-api-access-g4drl\") pod \"sub-test-model-9ca1fc-kserve-676778679c-7zdbj\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") " pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.448184 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.448112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:34.721529 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:34.721498    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"]
Sep 08 15:39:34.722909 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:39:34.722881    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc56971b2_107d_47bd_807c_208c709b2293.slice/crio-07e8979f1b9b839bbf75841478292934057cc49a25793c64c46ca10bf152deda WatchSource:0}: Error finding container 07e8979f1b9b839bbf75841478292934057cc49a25793c64c46ca10bf152deda: Status 404 returned error can't find the container with id 07e8979f1b9b839bbf75841478292934057cc49a25793c64c46ca10bf152deda
Sep 08 15:39:35.442745 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:35.442700    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96a211d-d7de-4bea-bdfe-586b204d562e" path="/var/lib/kubelet/pods/f96a211d-d7de-4bea-bdfe-586b204d562e/volumes"
Sep 08 15:39:35.678111 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:35.678080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj" event={"ID":"c56971b2-107d-47bd-807c-208c709b2293","Type":"ContainerStarted","Data":"82792466198a1abc9af351f53397d96913e5ca4b93757d7faf4df56e8ea8b006"}
Sep 08 15:39:35.678235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:35.678121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj" event={"ID":"c56971b2-107d-47bd-807c-208c709b2293","Type":"ContainerStarted","Data":"07e8979f1b9b839bbf75841478292934057cc49a25793c64c46ca10bf152deda"}
Sep 08 15:39:35.702688 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:35.702612    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj" podStartSLOduration=1.7025986770000001 podStartE2EDuration="1.702598677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:39:35.700727387 +0000 UTC m=+1324.776019696" watchObservedRunningTime="2026-09-08 15:39:35.702598677 +0000 UTC m=+1324.777890990"
Sep 08 15:39:38.205422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:38.205387    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:39:44.448756 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:44.448718    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:44.449264 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:44.448773    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:44.461425 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:44.461395    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:39:44.730296 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:39:44.730220    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:40:25.114993 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.114955    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"]
Sep 08 15:40:25.120614 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.120583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.127297 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.127273    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"kube-root-ca.crt\""
Sep 08 15:40:25.127297 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.127291    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"openshift-service-ca.crt\""
Sep 08 15:40:25.127807 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.127788    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"sub-test-model-9fb31f-kserve-self-signed-certs\""
Sep 08 15:40:25.131062 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.131038    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"default-dockercfg-9tg9m\""
Sep 08 15:40:25.148690 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.148663    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"]
Sep 08 15:40:25.213543 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.213522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-home\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.213663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.213551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-dshm\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.213663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.213608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrcl\" (UniqueName: \"kubernetes.io/projected/5b1f18ef-260c-441c-a393-461c200a8d54-kube-api-access-jtrcl\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.213663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.213630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-model-cache\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.213663 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.213648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1f18ef-260c-441c-a393-461c200a8d54-tls-certs\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.213795 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.213735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-tmp-dir\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314401 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-tmp-dir\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-home\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-dshm\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrcl\" (UniqueName: \"kubernetes.io/projected/5b1f18ef-260c-441c-a393-461c200a8d54-kube-api-access-jtrcl\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-model-cache\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314515 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1f18ef-260c-441c-a393-461c200a8d54-tls-certs\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314775 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-tmp-dir\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314872 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-home\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.314971 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.314951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-model-cache\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.316858 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.316821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-dshm\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.317103 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.317087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1f18ef-260c-441c-a393-461c200a8d54-tls-certs\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.325208 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.325186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrcl\" (UniqueName: \"kubernetes.io/projected/5b1f18ef-260c-441c-a393-461c200a8d54-kube-api-access-jtrcl\") pod \"sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.433077 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.433057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:25.710260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.710218    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"]
Sep 08 15:40:25.712567 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:40:25.712533    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b1f18ef_260c_441c_a393_461c200a8d54.slice/crio-e75e411fdb3a8c5599a77ca55bb7dc3672419d37bde25b67d33543803295de31 WatchSource:0}: Error finding container e75e411fdb3a8c5599a77ca55bb7dc3672419d37bde25b67d33543803295de31: Status 404 returned error can't find the container with id e75e411fdb3a8c5599a77ca55bb7dc3672419d37bde25b67d33543803295de31
Sep 08 15:40:25.921592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.921555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx" event={"ID":"5b1f18ef-260c-441c-a393-461c200a8d54","Type":"ContainerStarted","Data":"d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce"}
Sep 08 15:40:25.921736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.921602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx" event={"ID":"5b1f18ef-260c-441c-a393-461c200a8d54","Type":"ContainerStarted","Data":"e75e411fdb3a8c5599a77ca55bb7dc3672419d37bde25b67d33543803295de31"}
Sep 08 15:40:25.942689 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:25.942636    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx" podStartSLOduration=0.942621077 podStartE2EDuration="942.621077ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:40:25.940985331 +0000 UTC m=+1375.016277641" watchObservedRunningTime="2026-09-08 15:40:25.942621077 +0000 UTC m=+1375.017913361"
Sep 08 15:40:29.412046 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:29.412007    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:40:35.433466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:35.433422    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:35.433466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:35.433480    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:35.446583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:35.446558    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:35.983075 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:35.983048    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:40:49.147829 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.147796    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"]
Sep 08 15:40:49.151066 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.151050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.153993 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.153974    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bea11f\""
Sep 08 15:40:49.165440 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.165418    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"]
Sep 08 15:40:49.202471 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.202449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.202570 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.202482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l978s\" (UniqueName: \"kubernetes.io/projected/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-kube-api-access-l978s\") pod \"payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.281228 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.281202    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"]
Sep 08 15:40:49.284463 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.284446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.289869 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.287650    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bea11f-dockercfg-pd2hd\""
Sep 08 15:40:49.297355 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.297327    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"]
Sep 08 15:40:49.303424 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.303402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.303508 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.303434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l978s\" (UniqueName: \"kubernetes.io/projected/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-kube-api-access-l978s\") pod \"payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.304086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.304068    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.315950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.315930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l978s\" (UniqueName: \"kubernetes.io/projected/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-kube-api-access-l978s\") pod \"payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.404583 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.404512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6l47\" (UniqueName: \"kubernetes.io/projected/56863f64-a7c1-4062-9dc0-c1616730b451-kube-api-access-z6l47\") pod \"payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") " pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.404702 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.404580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56863f64-a7c1-4062-9dc0-c1616730b451-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") " pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.461031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.461006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:49.506045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.506012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56863f64-a7c1-4062-9dc0-c1616730b451-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") " pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.506200 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.506143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6l47\" (UniqueName: \"kubernetes.io/projected/56863f64-a7c1-4062-9dc0-c1616730b451-kube-api-access-z6l47\") pod \"payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") " pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.506909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.506883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56863f64-a7c1-4062-9dc0-c1616730b451-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") " pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.516250 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.516219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6l47\" (UniqueName: \"kubernetes.io/projected/56863f64-a7c1-4062-9dc0-c1616730b451-kube-api-access-z6l47\") pod \"payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") " pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.602106 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.602080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:49.730408 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.730379    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"]
Sep 08 15:40:49.731756 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:40:49.731732    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffa153f_28c2_42ac_82b6_ff1b1082b2e6.slice/crio-d4bb8c742850133fa1d50794e3de9dfcd7cb47ba2985fce056c5210bf6c45bee WatchSource:0}: Error finding container d4bb8c742850133fa1d50794e3de9dfcd7cb47ba2985fce056c5210bf6c45bee: Status 404 returned error can't find the container with id d4bb8c742850133fa1d50794e3de9dfcd7cb47ba2985fce056c5210bf6c45bee
Sep 08 15:40:49.874821 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:49.874780    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"]
Sep 08 15:40:49.875613 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:40:49.875587    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56863f64_a7c1_4062_9dc0_c1616730b451.slice/crio-652c39b794d3920abd4d3393e1abc19eac1f29415d07fe3c69a79336e2b0d498 WatchSource:0}: Error finding container 652c39b794d3920abd4d3393e1abc19eac1f29415d07fe3c69a79336e2b0d498: Status 404 returned error can't find the container with id 652c39b794d3920abd4d3393e1abc19eac1f29415d07fe3c69a79336e2b0d498
Sep 08 15:40:50.041271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.041169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2" event={"ID":"56863f64-a7c1-4062-9dc0-c1616730b451","Type":"ContainerStarted","Data":"ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1"}
Sep 08 15:40:50.041271 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.041256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2" event={"ID":"56863f64-a7c1-4062-9dc0-c1616730b451","Type":"ContainerStarted","Data":"652c39b794d3920abd4d3393e1abc19eac1f29415d07fe3c69a79336e2b0d498"}
Sep 08 15:40:50.041600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.041292    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:40:50.042854 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.042827    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb" event={"ID":"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6","Type":"ContainerStarted","Data":"e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5"}
Sep 08 15:40:50.042992 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.042886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb" event={"ID":"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6","Type":"ContainerStarted","Data":"d4bb8c742850133fa1d50794e3de9dfcd7cb47ba2985fce056c5210bf6c45bee"}
Sep 08 15:40:50.042992 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.042917    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:40:50.062249 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.062207    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2" podStartSLOduration=1.06219301 podStartE2EDuration="1.06219301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:40:50.060606791 +0000 UTC m=+1399.135899085" watchObservedRunningTime="2026-09-08 15:40:50.06219301 +0000 UTC m=+1399.137485302"
Sep 08 15:40:50.077771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:50.077729    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb" podStartSLOduration=1.077715432 podStartE2EDuration="1.077715432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:40:50.077163774 +0000 UTC m=+1399.152456085" watchObservedRunningTime="2026-09-08 15:40:50.077715432 +0000 UTC m=+1399.153007763"
Sep 08 15:40:58.118143 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:40:58.118107    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:41:01.052067 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:01.052024    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:41:01.052483 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:01.052198    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:41:17.329186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:17.329150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:41:26.136258 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:26.136182    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"]
Sep 08 15:41:27.275165 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:27.275133    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"]
Sep 08 15:41:31.104379 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:31.104345    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:41:41.156359 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.156309    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj" podUID="c56971b2-107d-47bd-807c-208c709b2293" containerName="main" containerID="cri-o://82792466198a1abc9af351f53397d96913e5ca4b93757d7faf4df56e8ea8b006" gracePeriod=45
Sep 08 15:41:41.313301 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.313265    2575 generic.go:356] "Generic (PLEG): container finished" podID="c56971b2-107d-47bd-807c-208c709b2293" containerID="82792466198a1abc9af351f53397d96913e5ca4b93757d7faf4df56e8ea8b006" exitCode=0
Sep 08 15:41:41.313436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.313334    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj" event={"ID":"c56971b2-107d-47bd-807c-208c709b2293","Type":"ContainerDied","Data":"82792466198a1abc9af351f53397d96913e5ca4b93757d7faf4df56e8ea8b006"}
Sep 08 15:41:41.404267 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.404245    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:41:41.532427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532361    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4drl\" (UniqueName: \"kubernetes.io/projected/c56971b2-107d-47bd-807c-208c709b2293-kube-api-access-g4drl\") pod \"c56971b2-107d-47bd-807c-208c709b2293\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") "
Sep 08 15:41:41.532427 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532404    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-dshm\") pod \"c56971b2-107d-47bd-807c-208c709b2293\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") "
Sep 08 15:41:41.532618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532442    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56971b2-107d-47bd-807c-208c709b2293-tls-certs\") pod \"c56971b2-107d-47bd-807c-208c709b2293\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") "
Sep 08 15:41:41.532618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532532    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-home\") pod \"c56971b2-107d-47bd-807c-208c709b2293\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") "
Sep 08 15:41:41.532618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-model-cache\") pod \"c56971b2-107d-47bd-807c-208c709b2293\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") "
Sep 08 15:41:41.532618 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532587    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-tmp-dir\") pod \"c56971b2-107d-47bd-807c-208c709b2293\" (UID: \"c56971b2-107d-47bd-807c-208c709b2293\") "
Sep 08 15:41:41.532925 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532835    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-model-cache" (OuterVolumeSpecName: "model-cache") pod "c56971b2-107d-47bd-807c-208c709b2293" (UID: "c56971b2-107d-47bd-807c-208c709b2293"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:41.532925 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-home" (OuterVolumeSpecName: "home") pod "c56971b2-107d-47bd-807c-208c709b2293" (UID: "c56971b2-107d-47bd-807c-208c709b2293"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:41.533115 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.532936    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c56971b2-107d-47bd-807c-208c709b2293" (UID: "c56971b2-107d-47bd-807c-208c709b2293"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:41.534779 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.534759    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56971b2-107d-47bd-807c-208c709b2293-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c56971b2-107d-47bd-807c-208c709b2293" (UID: "c56971b2-107d-47bd-807c-208c709b2293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:41:41.535202 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.535180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-dshm" (OuterVolumeSpecName: "dshm") pod "c56971b2-107d-47bd-807c-208c709b2293" (UID: "c56971b2-107d-47bd-807c-208c709b2293"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:41.535260 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.535183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56971b2-107d-47bd-807c-208c709b2293-kube-api-access-g4drl" (OuterVolumeSpecName: "kube-api-access-g4drl") pod "c56971b2-107d-47bd-807c-208c709b2293" (UID: "c56971b2-107d-47bd-807c-208c709b2293"). InnerVolumeSpecName "kube-api-access-g4drl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:41.633303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.633279    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4drl\" (UniqueName: \"kubernetes.io/projected/c56971b2-107d-47bd-807c-208c709b2293-kube-api-access-g4drl\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:41.633303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.633300    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:41.633436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.633310    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56971b2-107d-47bd-807c-208c709b2293-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:41.633436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.633318    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:41.633436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.633325    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:41.633436 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:41.633333    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56971b2-107d-47bd-807c-208c709b2293-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.295674 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.295633    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx" podUID="5b1f18ef-260c-441c-a393-461c200a8d54" containerName="main" containerID="cri-o://d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce" gracePeriod=45
Sep 08 15:41:42.320196 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.320171    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"
Sep 08 15:41:42.320303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.320188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj" event={"ID":"c56971b2-107d-47bd-807c-208c709b2293","Type":"ContainerDied","Data":"07e8979f1b9b839bbf75841478292934057cc49a25793c64c46ca10bf152deda"}
Sep 08 15:41:42.320303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.320247    2575 scope.go:117] "RemoveContainer" containerID="82792466198a1abc9af351f53397d96913e5ca4b93757d7faf4df56e8ea8b006"
Sep 08 15:41:42.369340 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.369315    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"]
Sep 08 15:41:42.373202 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.373181    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ca1fc/sub-test-model-9ca1fc-kserve-676778679c-7zdbj"]
Sep 08 15:41:42.539795 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.539766    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:41:42.641222 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641194    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtrcl\" (UniqueName: \"kubernetes.io/projected/5b1f18ef-260c-441c-a393-461c200a8d54-kube-api-access-jtrcl\") pod \"5b1f18ef-260c-441c-a393-461c200a8d54\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") "
Sep 08 15:41:42.641375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641237    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-home\") pod \"5b1f18ef-260c-441c-a393-461c200a8d54\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") "
Sep 08 15:41:42.641375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641285    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-dshm\") pod \"5b1f18ef-260c-441c-a393-461c200a8d54\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") "
Sep 08 15:41:42.641375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641337    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-tmp-dir\") pod \"5b1f18ef-260c-441c-a393-461c200a8d54\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") "
Sep 08 15:41:42.641375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641364    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-model-cache\") pod \"5b1f18ef-260c-441c-a393-461c200a8d54\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") "
Sep 08 15:41:42.641586 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641403    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1f18ef-260c-441c-a393-461c200a8d54-tls-certs\") pod \"5b1f18ef-260c-441c-a393-461c200a8d54\" (UID: \"5b1f18ef-260c-441c-a393-461c200a8d54\") "
Sep 08 15:41:42.641637 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641572    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b1f18ef-260c-441c-a393-461c200a8d54" (UID: "5b1f18ef-260c-441c-a393-461c200a8d54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:42.641637 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641579    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-home" (OuterVolumeSpecName: "home") pod "5b1f18ef-260c-441c-a393-461c200a8d54" (UID: "5b1f18ef-260c-441c-a393-461c200a8d54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:42.641727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641650    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b1f18ef-260c-441c-a393-461c200a8d54" (UID: "5b1f18ef-260c-441c-a393-461c200a8d54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:42.641780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641749    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.641780 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641769    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.641873 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.641780    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.643523 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.643503    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-dshm" (OuterVolumeSpecName: "dshm") pod "5b1f18ef-260c-441c-a393-461c200a8d54" (UID: "5b1f18ef-260c-441c-a393-461c200a8d54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:41:42.643586 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.643566    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1f18ef-260c-441c-a393-461c200a8d54-kube-api-access-jtrcl" (OuterVolumeSpecName: "kube-api-access-jtrcl") pod "5b1f18ef-260c-441c-a393-461c200a8d54" (UID: "5b1f18ef-260c-441c-a393-461c200a8d54"). InnerVolumeSpecName "kube-api-access-jtrcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:42.643630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.643609    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1f18ef-260c-441c-a393-461c200a8d54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b1f18ef-260c-441c-a393-461c200a8d54" (UID: "5b1f18ef-260c-441c-a393-461c200a8d54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:41:42.743002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.742968    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1f18ef-260c-441c-a393-461c200a8d54-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.743002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.743000    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrcl\" (UniqueName: \"kubernetes.io/projected/5b1f18ef-260c-441c-a393-461c200a8d54-kube-api-access-jtrcl\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.743002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:42.743010    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1f18ef-260c-441c-a393-461c200a8d54-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:43.327129 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.327096    2575 generic.go:356] "Generic (PLEG): container finished" podID="5b1f18ef-260c-441c-a393-461c200a8d54" containerID="d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce" exitCode=0
Sep 08 15:41:43.327581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.327140    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"
Sep 08 15:41:43.327581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.327189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx" event={"ID":"5b1f18ef-260c-441c-a393-461c200a8d54","Type":"ContainerDied","Data":"d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce"}
Sep 08 15:41:43.327581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.327234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx" event={"ID":"5b1f18ef-260c-441c-a393-461c200a8d54","Type":"ContainerDied","Data":"e75e411fdb3a8c5599a77ca55bb7dc3672419d37bde25b67d33543803295de31"}
Sep 08 15:41:43.327581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.327255    2575 scope.go:117] "RemoveContainer" containerID="d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce"
Sep 08 15:41:43.336861 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.336816    2575 scope.go:117] "RemoveContainer" containerID="d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce"
Sep 08 15:41:43.337161 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:41:43.337143    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce\": container with ID starting with d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce not found: ID does not exist" containerID="d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce"
Sep 08 15:41:43.337232 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.337171    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce"} err="failed to get container status \"d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce\": rpc error: code = NotFound desc = could not find container \"d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce\": container with ID starting with d2c86bb3fd539415e101efa59bfb0318f95472960a463fa6687f9aa5b6e903ce not found: ID does not exist"
Sep 08 15:41:43.350837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.350810    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"]
Sep 08 15:41:43.354938 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.354913    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/sub-test-model-9fb31f-kserve-59c89b6ffc-2vnlx"]
Sep 08 15:41:43.441592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.441560    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1f18ef-260c-441c-a393-461c200a8d54" path="/var/lib/kubelet/pods/5b1f18ef-260c-441c-a393-461c200a8d54/volumes"
Sep 08 15:41:43.442073 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:43.442048    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56971b2-107d-47bd-807c-208c709b2293" path="/var/lib/kubelet/pods/c56971b2-107d-47bd-807c-208c709b2293/volumes"
Sep 08 15:41:50.751156 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:50.751122    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:41:53.888018 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:53.887984    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"]
Sep 08 15:41:53.888409 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:53.888264    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2" podUID="56863f64-a7c1-4062-9dc0-c1616730b451" containerName="payload-processing" containerID="cri-o://ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1" gracePeriod=30
Sep 08 15:41:53.897096 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:53.897058    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"]
Sep 08 15:41:53.897299 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:53.897274    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb" podUID="0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" containerName="payload-pre-processing" containerID="cri-o://e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5" gracePeriod=30
Sep 08 15:41:54.160367 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.160346    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:41:54.162911 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.162895    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:41:54.233186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233155    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-plugins-config-volume\") pod \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") "
Sep 08 15:41:54.233457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233192    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56863f64-a7c1-4062-9dc0-c1616730b451-plugins-config-volume\") pod \"56863f64-a7c1-4062-9dc0-c1616730b451\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") "
Sep 08 15:41:54.233457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233227    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6l47\" (UniqueName: \"kubernetes.io/projected/56863f64-a7c1-4062-9dc0-c1616730b451-kube-api-access-z6l47\") pod \"56863f64-a7c1-4062-9dc0-c1616730b451\" (UID: \"56863f64-a7c1-4062-9dc0-c1616730b451\") "
Sep 08 15:41:54.233457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233279    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l978s\" (UniqueName: \"kubernetes.io/projected/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-kube-api-access-l978s\") pod \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\" (UID: \"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6\") "
Sep 08 15:41:54.233650 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233567    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" (UID: "0ffa153f-28c2-42ac-82b6-ff1b1082b2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:41:54.233650 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233634    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:54.233965 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.233934    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56863f64-a7c1-4062-9dc0-c1616730b451-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56863f64-a7c1-4062-9dc0-c1616730b451" (UID: "56863f64-a7c1-4062-9dc0-c1616730b451"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:41:54.235814 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.235786    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-kube-api-access-l978s" (OuterVolumeSpecName: "kube-api-access-l978s") pod "0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" (UID: "0ffa153f-28c2-42ac-82b6-ff1b1082b2e6"). InnerVolumeSpecName "kube-api-access-l978s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:54.236104 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.236083    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56863f64-a7c1-4062-9dc0-c1616730b451-kube-api-access-z6l47" (OuterVolumeSpecName: "kube-api-access-z6l47") pod "56863f64-a7c1-4062-9dc0-c1616730b451" (UID: "56863f64-a7c1-4062-9dc0-c1616730b451"). InnerVolumeSpecName "kube-api-access-z6l47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:54.334810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.334779    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l978s\" (UniqueName: \"kubernetes.io/projected/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6-kube-api-access-l978s\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:54.334810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.334807    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56863f64-a7c1-4062-9dc0-c1616730b451-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:54.334974 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.334822    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6l47\" (UniqueName: \"kubernetes.io/projected/56863f64-a7c1-4062-9dc0-c1616730b451-kube-api-access-z6l47\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:41:54.389212 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.389182    2575 generic.go:356] "Generic (PLEG): container finished" podID="56863f64-a7c1-4062-9dc0-c1616730b451" containerID="ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1" exitCode=0
Sep 08 15:41:54.389324 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.389233    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"
Sep 08 15:41:54.389324 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.389271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2" event={"ID":"56863f64-a7c1-4062-9dc0-c1616730b451","Type":"ContainerDied","Data":"ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1"}
Sep 08 15:41:54.389417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.389336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2" event={"ID":"56863f64-a7c1-4062-9dc0-c1616730b451","Type":"ContainerDied","Data":"652c39b794d3920abd4d3393e1abc19eac1f29415d07fe3c69a79336e2b0d498"}
Sep 08 15:41:54.389417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.389366    2575 scope.go:117] "RemoveContainer" containerID="ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1"
Sep 08 15:41:54.390730 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.390708    2575 generic.go:356] "Generic (PLEG): container finished" podID="0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" containerID="e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5" exitCode=0
Sep 08 15:41:54.390860 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.390753    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"
Sep 08 15:41:54.390860 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.390801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb" event={"ID":"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6","Type":"ContainerDied","Data":"e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5"}
Sep 08 15:41:54.390958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.390840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb" event={"ID":"0ffa153f-28c2-42ac-82b6-ff1b1082b2e6","Type":"ContainerDied","Data":"d4bb8c742850133fa1d50794e3de9dfcd7cb47ba2985fce056c5210bf6c45bee"}
Sep 08 15:41:54.400303 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.400285    2575 scope.go:117] "RemoveContainer" containerID="ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1"
Sep 08 15:41:54.400601 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:41:54.400576    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1\": container with ID starting with ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1 not found: ID does not exist" containerID="ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1"
Sep 08 15:41:54.400672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.400612    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1"} err="failed to get container status \"ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1\": rpc error: code = NotFound desc = could not find container \"ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1\": container with ID starting with ae5621c3597fd72fa6beadc81a9075f99b16c17791d7f5cfe3fd019ed40594b1 not found: ID does not exist"
Sep 08 15:41:54.400672 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.400635    2575 scope.go:117] "RemoveContainer" containerID="e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5"
Sep 08 15:41:54.408336 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.408316    2575 scope.go:117] "RemoveContainer" containerID="e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5"
Sep 08 15:41:54.408594 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:41:54.408577    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5\": container with ID starting with e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5 not found: ID does not exist" containerID="e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5"
Sep 08 15:41:54.408665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.408605    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5"} err="failed to get container status \"e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5\": rpc error: code = NotFound desc = could not find container \"e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5\": container with ID starting with e6be9ac551339ced42eb79be9a6b6cd0677156bd9626dc2c046cb0119c384ae5 not found: ID does not exist"
Sep 08 15:41:54.416990 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.416966    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"]
Sep 08 15:41:54.421336 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.421316    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bea11f-bf956d5f8-qwcjb"]
Sep 08 15:41:54.437829 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.437804    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"]
Sep 08 15:41:54.449622 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:54.449601    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bea11f-f5cfd79b5-trzz2"]
Sep 08 15:41:55.441645 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:55.441616    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" path="/var/lib/kubelet/pods/0ffa153f-28c2-42ac-82b6-ff1b1082b2e6/volumes"
Sep 08 15:41:55.442011 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:41:55.441965    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56863f64-a7c1-4062-9dc0-c1616730b451" path="/var/lib/kubelet/pods/56863f64-a7c1-4062-9dc0-c1616730b451/volumes"
Sep 08 15:42:07.016231 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:07.016181    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:42:11.885386 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:11.885358    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"]
Sep 08 15:42:11.885872 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:11.885597    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r" podUID="6f80f3ae-ad7a-43d0-98b8-f052f27690ad" containerName="payload-processing" containerID="cri-o://eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf" gracePeriod=30
Sep 08 15:42:12.141113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.141047    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:42:12.274531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.274499    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdwnr\" (UniqueName: \"kubernetes.io/projected/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-kube-api-access-hdwnr\") pod \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") "
Sep 08 15:42:12.274677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.274539    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-plugins-config-volume\") pod \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\" (UID: \"6f80f3ae-ad7a-43d0-98b8-f052f27690ad\") "
Sep 08 15:42:12.274923 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.274897    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f80f3ae-ad7a-43d0-98b8-f052f27690ad" (UID: "6f80f3ae-ad7a-43d0-98b8-f052f27690ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:42:12.276860 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.276818    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-kube-api-access-hdwnr" (OuterVolumeSpecName: "kube-api-access-hdwnr") pod "6f80f3ae-ad7a-43d0-98b8-f052f27690ad" (UID: "6f80f3ae-ad7a-43d0-98b8-f052f27690ad"). InnerVolumeSpecName "kube-api-access-hdwnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:42:12.375171 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.375145    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdwnr\" (UniqueName: \"kubernetes.io/projected/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-kube-api-access-hdwnr\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:12.375171 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.375166    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f80f3ae-ad7a-43d0-98b8-f052f27690ad-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:12.484806 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.484731    2575 generic.go:356] "Generic (PLEG): container finished" podID="6f80f3ae-ad7a-43d0-98b8-f052f27690ad" containerID="eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf" exitCode=0
Sep 08 15:42:12.484806 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.484766    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"
Sep 08 15:42:12.484991 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.484815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r" event={"ID":"6f80f3ae-ad7a-43d0-98b8-f052f27690ad","Type":"ContainerDied","Data":"eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf"}
Sep 08 15:42:12.484991 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.484880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r" event={"ID":"6f80f3ae-ad7a-43d0-98b8-f052f27690ad","Type":"ContainerDied","Data":"ccd9f1be782223e541b9e15dd2be5db4596f35cc1a1b918a9e6707bf5bb8bba1"}
Sep 08 15:42:12.484991 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.484902    2575 scope.go:117] "RemoveContainer" containerID="eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf"
Sep 08 15:42:12.494245 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.494229    2575 scope.go:117] "RemoveContainer" containerID="eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf"
Sep 08 15:42:12.494506 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:42:12.494491    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf\": container with ID starting with eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf not found: ID does not exist" containerID="eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf"
Sep 08 15:42:12.494553 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.494513    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf"} err="failed to get container status \"eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf\": rpc error: code = NotFound desc = could not find container \"eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf\": container with ID starting with eb1160e9fbdc8d6fa3216bddda76dbf5ea13c83fcba5acba7f5ae15c853e4fbf not found: ID does not exist"
Sep 08 15:42:12.509612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.509589    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"]
Sep 08 15:42:12.516728 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:12.516706    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-03e8df-57b9f4d96f-28h5r"]
Sep 08 15:42:13.441816 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:13.441771    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f80f3ae-ad7a-43d0-98b8-f052f27690ad" path="/var/lib/kubelet/pods/6f80f3ae-ad7a-43d0-98b8-f052f27690ad/volumes"
Sep 08 15:42:18.209865 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:18.209812    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:42:30.237502 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.237461    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"]
Sep 08 15:42:30.238008 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.237715    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" podUID="70b63a8a-2ea3-4317-94bc-248015e992ca" containerName="maas-api" containerID="cri-o://05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078" gracePeriod=30
Sep 08 15:42:30.345528 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.345463    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"]
Sep 08 15:42:30.346190 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.345785    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7" podUID="e70c1d36-888f-44a6-971c-8512b9ad5b9e" containerName="payload-pre-processing" containerID="cri-o://7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e" gracePeriod=30
Sep 08 15:42:30.503138 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.503073    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:42:30.528391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.528367    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v78rs\" (UniqueName: \"kubernetes.io/projected/70b63a8a-2ea3-4317-94bc-248015e992ca-kube-api-access-v78rs\") pod \"70b63a8a-2ea3-4317-94bc-248015e992ca\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") "
Sep 08 15:42:30.528505 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.528434    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b63a8a-2ea3-4317-94bc-248015e992ca-openshift-service-ca\") pod \"70b63a8a-2ea3-4317-94bc-248015e992ca\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") "
Sep 08 15:42:30.528505 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.528495    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-metrics-certs\") pod \"70b63a8a-2ea3-4317-94bc-248015e992ca\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") "
Sep 08 15:42:30.528628 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.528519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls\") pod \"70b63a8a-2ea3-4317-94bc-248015e992ca\" (UID: \"70b63a8a-2ea3-4317-94bc-248015e992ca\") "
Sep 08 15:42:30.528836 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.528804    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b63a8a-2ea3-4317-94bc-248015e992ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70b63a8a-2ea3-4317-94bc-248015e992ca" (UID: "70b63a8a-2ea3-4317-94bc-248015e992ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:42:30.531105 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.531078    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70b63a8a-2ea3-4317-94bc-248015e992ca" (UID: "70b63a8a-2ea3-4317-94bc-248015e992ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:42:30.531395 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.531372    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b63a8a-2ea3-4317-94bc-248015e992ca-kube-api-access-v78rs" (OuterVolumeSpecName: "kube-api-access-v78rs") pod "70b63a8a-2ea3-4317-94bc-248015e992ca" (UID: "70b63a8a-2ea3-4317-94bc-248015e992ca"). InnerVolumeSpecName "kube-api-access-v78rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:42:30.532033 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.531998    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70b63a8a-2ea3-4317-94bc-248015e992ca" (UID: "70b63a8a-2ea3-4317-94bc-248015e992ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:42:30.573486 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.573456    2575 generic.go:356] "Generic (PLEG): container finished" podID="e70c1d36-888f-44a6-971c-8512b9ad5b9e" containerID="7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e" exitCode=0
Sep 08 15:42:30.573603 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.573546    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7" event={"ID":"e70c1d36-888f-44a6-971c-8512b9ad5b9e","Type":"ContainerDied","Data":"7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e"}
Sep 08 15:42:30.574917 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.574893    2575 generic.go:356] "Generic (PLEG): container finished" podID="70b63a8a-2ea3-4317-94bc-248015e992ca" containerID="05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078" exitCode=0
Sep 08 15:42:30.575020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.574938    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"
Sep 08 15:42:30.575020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.574968    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" event={"ID":"70b63a8a-2ea3-4317-94bc-248015e992ca","Type":"ContainerDied","Data":"05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078"}
Sep 08 15:42:30.575020 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.575011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl" event={"ID":"70b63a8a-2ea3-4317-94bc-248015e992ca","Type":"ContainerDied","Data":"c50571582b3b6ba9894e75865eb086b24d6b811e2bedc4ffbad08d8820f110cb"}
Sep 08 15:42:30.575148 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.575038    2575 scope.go:117] "RemoveContainer" containerID="05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078"
Sep 08 15:42:30.584083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.584066    2575 scope.go:117] "RemoveContainer" containerID="05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078"
Sep 08 15:42:30.584386 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:42:30.584366    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078\": container with ID starting with 05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078 not found: ID does not exist" containerID="05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078"
Sep 08 15:42:30.584453 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.584394    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078"} err="failed to get container status \"05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078\": rpc error: code = NotFound desc = could not find container \"05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078\": container with ID starting with 05a2285b4757f9b989b77e47a3e5c1f7381e37e63fbfde5650911b6af5a2a078 not found: ID does not exist"
Sep 08 15:42:30.593267 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.593251    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:42:30.604755 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.604735    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"]
Sep 08 15:42:30.607541 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607518    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"]
Sep 08 15:42:30.607837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607824    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56863f64-a7c1-4062-9dc0-c1616730b451" containerName="payload-processing"
Sep 08 15:42:30.607837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607837    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56863f64-a7c1-4062-9dc0-c1616730b451" containerName="payload-processing"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607867    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b63a8a-2ea3-4317-94bc-248015e992ca" containerName="maas-api"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607873    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b63a8a-2ea3-4317-94bc-248015e992ca" containerName="maas-api"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607879    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1f18ef-260c-441c-a393-461c200a8d54" containerName="main"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607884    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1f18ef-260c-441c-a393-461c200a8d54" containerName="main"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607892    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f80f3ae-ad7a-43d0-98b8-f052f27690ad" containerName="payload-processing"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607897    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f80f3ae-ad7a-43d0-98b8-f052f27690ad" containerName="payload-processing"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607903    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56971b2-107d-47bd-807c-208c709b2293" containerName="main"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607907    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56971b2-107d-47bd-807c-208c709b2293" containerName="main"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607916    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70c1d36-888f-44a6-971c-8512b9ad5b9e" containerName="payload-pre-processing"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607921    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70c1d36-888f-44a6-971c-8512b9ad5b9e" containerName="payload-pre-processing"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607926    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" containerName="payload-pre-processing"
Sep 08 15:42:30.607935 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607931    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" containerName="payload-pre-processing"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607976    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f80f3ae-ad7a-43d0-98b8-f052f27690ad" containerName="payload-processing"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607985    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b63a8a-2ea3-4317-94bc-248015e992ca" containerName="maas-api"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607990    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1f18ef-260c-441c-a393-461c200a8d54" containerName="main"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.607995    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56863f64-a7c1-4062-9dc0-c1616730b451" containerName="payload-processing"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.608000    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70c1d36-888f-44a6-971c-8512b9ad5b9e" containerName="payload-pre-processing"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.608004    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ffa153f-28c2-42ac-82b6-ff1b1082b2e6" containerName="payload-pre-processing"
Sep 08 15:42:30.608290 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.608011    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56971b2-107d-47bd-807c-208c709b2293" containerName="main"
Sep 08 15:42:30.610603 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.610589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.614244 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.614208    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"openshift-service-ca.crt\""
Sep 08 15:42:30.614244 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.614222    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"kube-root-ca.crt\""
Sep 08 15:42:30.614244 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.614229    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"default-dockercfg-9tg9m\""
Sep 08 15:42:30.614519 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.614223    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9fb31f\"/\"rate-test-model-9fb31f-kserve-self-signed-certs\""
Sep 08 15:42:30.620577 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.620559    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a7ae4e-56d44d5bc6-bf6gl"]
Sep 08 15:42:30.623901 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.623881    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"]
Sep 08 15:42:30.629555 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629537    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkc6l\" (UniqueName: \"kubernetes.io/projected/e70c1d36-888f-44a6-971c-8512b9ad5b9e-kube-api-access-tkc6l\") pod \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") "
Sep 08 15:42:30.629640 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629585    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70c1d36-888f-44a6-971c-8512b9ad5b9e-plugins-config-volume\") pod \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\" (UID: \"e70c1d36-888f-44a6-971c-8512b9ad5b9e\") "
Sep 08 15:42:30.629681 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-dshm\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.629732 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tmp-dir\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.629732 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tls-certs\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.629887 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-model-cache\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.629954 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6hd\" (UniqueName: \"kubernetes.io/projected/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-kube-api-access-xp6hd\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.630008 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.629955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-home\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.630066 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.630041    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70c1d36-888f-44a6-971c-8512b9ad5b9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e70c1d36-888f-44a6-971c-8512b9ad5b9e" (UID: "e70c1d36-888f-44a6-971c-8512b9ad5b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:42:30.630113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.630059    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v78rs\" (UniqueName: \"kubernetes.io/projected/70b63a8a-2ea3-4317-94bc-248015e992ca-kube-api-access-v78rs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:30.630113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.630079    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b63a8a-2ea3-4317-94bc-248015e992ca-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:30.630113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.630090    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:30.630113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.630100    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b63a8a-2ea3-4317-94bc-248015e992ca-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:30.631627 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.631608    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70c1d36-888f-44a6-971c-8512b9ad5b9e-kube-api-access-tkc6l" (OuterVolumeSpecName: "kube-api-access-tkc6l") pod "e70c1d36-888f-44a6-971c-8512b9ad5b9e" (UID: "e70c1d36-888f-44a6-971c-8512b9ad5b9e"). InnerVolumeSpecName "kube-api-access-tkc6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:42:30.730951 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.730923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6hd\" (UniqueName: \"kubernetes.io/projected/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-kube-api-access-xp6hd\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.730959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-home\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-dshm\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tmp-dir\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tls-certs\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731317 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-model-cache\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731317 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731214    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70c1d36-888f-44a6-971c-8512b9ad5b9e-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:30.731317 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731231    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkc6l\" (UniqueName: \"kubernetes.io/projected/e70c1d36-888f-44a6-971c-8512b9ad5b9e-kube-api-access-tkc6l\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:42:30.731469 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-home\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731469 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tmp-dir\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.731597 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.731578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-model-cache\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.733438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.733417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-dshm\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.733670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.733654    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tls-certs\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.740368 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.740348    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6hd\" (UniqueName: \"kubernetes.io/projected/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-kube-api-access-xp6hd\") pod \"rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") " pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:30.922063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:30.922039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:31.198899 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.198870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"]
Sep 08 15:42:31.200918 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:42:31.200893    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode46c7178_65f1_43d3_9b80_5dfe2afbafc9.slice/crio-939ee21c8549c6eb12b0207986cc89c56b826f5e0234d88f13425496aec04fd2 WatchSource:0}: Error finding container 939ee21c8549c6eb12b0207986cc89c56b826f5e0234d88f13425496aec04fd2: Status 404 returned error can't find the container with id 939ee21c8549c6eb12b0207986cc89c56b826f5e0234d88f13425496aec04fd2
Sep 08 15:42:31.441477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.441446    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b63a8a-2ea3-4317-94bc-248015e992ca" path="/var/lib/kubelet/pods/70b63a8a-2ea3-4317-94bc-248015e992ca/volumes"
Sep 08 15:42:31.577350 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.577322    2575 scope.go:117] "RemoveContainer" containerID="7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e"
Sep 08 15:42:31.581113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.581085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9" event={"ID":"e46c7178-65f1-43d3-9b80-5dfe2afbafc9","Type":"ContainerStarted","Data":"28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7"}
Sep 08 15:42:31.581241 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.581124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9" event={"ID":"e46c7178-65f1-43d3-9b80-5dfe2afbafc9","Type":"ContainerStarted","Data":"939ee21c8549c6eb12b0207986cc89c56b826f5e0234d88f13425496aec04fd2"}
Sep 08 15:42:31.582519 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.582491    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"
Sep 08 15:42:31.582642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.582525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7" event={"ID":"e70c1d36-888f-44a6-971c-8512b9ad5b9e","Type":"ContainerDied","Data":"3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1"}
Sep 08 15:42:31.582642 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.582584    2575 scope.go:117] "RemoveContainer" containerID="7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e"
Sep 08 15:42:31.587208 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:42:31.587178    2575 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-pre-processing_payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7_openshift-ingress_e70c1d36-888f-44a6-971c-8512b9ad5b9e_0 in pod sandbox 3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1 from index: no such id: '7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e'" containerID="7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e"
Sep 08 15:42:31.587325 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:42:31.587224    2575 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-pre-processing_payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7_openshift-ingress_e70c1d36-888f-44a6-971c-8512b9ad5b9e_0 in pod sandbox 3844f90d64f6ca09f65f391d7862bfff79e55a26897fe07293dedc239dcb5ad1 from index: no such id: '7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e'" containerID="7a1c229f58a80d8c9843bf72b118048d7229079f9098e0b56d8b8e533c5d698e"
Sep 08 15:42:31.604955 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.604911    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9" podStartSLOduration=1.604893847 podStartE2EDuration="1.604893847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:42:31.60297577 +0000 UTC m=+1500.678268077" watchObservedRunningTime="2026-09-08 15:42:31.604893847 +0000 UTC m=+1500.680186141"
Sep 08 15:42:31.619439 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.619419    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"]
Sep 08 15:42:31.625147 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:31.625126    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a7ae4e-6865bd549d-g8cm7"]
Sep 08 15:42:33.441955 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:33.441911    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70c1d36-888f-44a6-971c-8512b9ad5b9e" path="/var/lib/kubelet/pods/e70c1d36-888f-44a6-971c-8512b9ad5b9e/volumes"
Sep 08 15:42:36.606737 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:36.606696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:42:40.922771 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:40.922734    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:40.923327 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:40.922784    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:40.935960 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:40.935930    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:41.644727 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:41.644695    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:42:51.995080 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:42:51.995041    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:43:01.913034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:01.912992    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:43:09.282688 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:09.282650    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"]
Sep 08 15:43:12.396122 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:12.396082    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:43:15.007457 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.007423    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"]
Sep 08 15:43:15.011065 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.011045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.013864 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.013825    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41de51\""
Sep 08 15:43:15.020707 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.020688    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"]
Sep 08 15:43:15.181308 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.181270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.181475 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.181349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4zb\" (UniqueName: \"kubernetes.io/projected/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-kube-api-access-jg4zb\") pod \"payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.282707 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.282641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.282707 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.282689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4zb\" (UniqueName: \"kubernetes.io/projected/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-kube-api-access-jg4zb\") pod \"payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.283389 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.283369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.291730 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.291705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4zb\" (UniqueName: \"kubernetes.io/projected/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-kube-api-access-jg4zb\") pod \"payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.321662 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.321643    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.593697 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.593633    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"]
Sep 08 15:43:15.596141 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:43:15.596111    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b31a9f_bd42_43d5_bebd_02e22ad37d34.slice/crio-7bd25efb0b7d645fbcbf6542c334e0c96c890b1abc656faef6ff407a08fd394b WatchSource:0}: Error finding container 7bd25efb0b7d645fbcbf6542c334e0c96c890b1abc656faef6ff407a08fd394b: Status 404 returned error can't find the container with id 7bd25efb0b7d645fbcbf6542c334e0c96c890b1abc656faef6ff407a08fd394b
Sep 08 15:43:15.793509 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.793480    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st" event={"ID":"f1b31a9f-bd42-43d5-bebd-02e22ad37d34","Type":"ContainerStarted","Data":"e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054"}
Sep 08 15:43:15.793665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.793528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st" event={"ID":"f1b31a9f-bd42-43d5-bebd-02e22ad37d34","Type":"ContainerStarted","Data":"7bd25efb0b7d645fbcbf6542c334e0c96c890b1abc656faef6ff407a08fd394b"}
Sep 08 15:43:15.793665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.793590    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:15.814129 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:15.814084    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st" podStartSLOduration=1.814071121 podStartE2EDuration="1.814071121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:43:15.812233113 +0000 UTC m=+1544.887525425" watchObservedRunningTime="2026-09-08 15:43:15.814071121 +0000 UTC m=+1544.889363413"
Sep 08 15:43:24.302614 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.302568    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9" podUID="e46c7178-65f1-43d3-9b80-5dfe2afbafc9" containerName="main" containerID="cri-o://28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7" gracePeriod=45
Sep 08 15:43:24.552806 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.552745    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:43:24.564286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564256    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tls-certs\") pod \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") "
Sep 08 15:43:24.564410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564314    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-model-cache\") pod \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") "
Sep 08 15:43:24.564410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564340    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tmp-dir\") pod \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") "
Sep 08 15:43:24.564410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564381    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp6hd\" (UniqueName: \"kubernetes.io/projected/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-kube-api-access-xp6hd\") pod \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") "
Sep 08 15:43:24.564410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564405    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-home\") pod \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") "
Sep 08 15:43:24.564615 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564428    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-dshm\") pod \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\" (UID: \"e46c7178-65f1-43d3-9b80-5dfe2afbafc9\") "
Sep 08 15:43:24.564615 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564494    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e46c7178-65f1-43d3-9b80-5dfe2afbafc9" (UID: "e46c7178-65f1-43d3-9b80-5dfe2afbafc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:43:24.564615 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564591    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e46c7178-65f1-43d3-9b80-5dfe2afbafc9" (UID: "e46c7178-65f1-43d3-9b80-5dfe2afbafc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:43:24.564769 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564726    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-home" (OuterVolumeSpecName: "home") pod "e46c7178-65f1-43d3-9b80-5dfe2afbafc9" (UID: "e46c7178-65f1-43d3-9b80-5dfe2afbafc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:43:24.564769 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564753    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:43:24.564876 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.564773    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:43:24.566365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.566345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e46c7178-65f1-43d3-9b80-5dfe2afbafc9" (UID: "e46c7178-65f1-43d3-9b80-5dfe2afbafc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:43:24.566553 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.566534    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-kube-api-access-xp6hd" (OuterVolumeSpecName: "kube-api-access-xp6hd") pod "e46c7178-65f1-43d3-9b80-5dfe2afbafc9" (UID: "e46c7178-65f1-43d3-9b80-5dfe2afbafc9"). InnerVolumeSpecName "kube-api-access-xp6hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:43:24.566703 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.566678    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-dshm" (OuterVolumeSpecName: "dshm") pod "e46c7178-65f1-43d3-9b80-5dfe2afbafc9" (UID: "e46c7178-65f1-43d3-9b80-5dfe2afbafc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:43:24.665519 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.665497    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp6hd\" (UniqueName: \"kubernetes.io/projected/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-kube-api-access-xp6hd\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:43:24.665519 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.665519    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:43:24.665667 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.665527    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:43:24.665667 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.665535    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46c7178-65f1-43d3-9b80-5dfe2afbafc9-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:43:24.836698 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.836632    2575 generic.go:356] "Generic (PLEG): container finished" podID="e46c7178-65f1-43d3-9b80-5dfe2afbafc9" containerID="28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7" exitCode=0
Sep 08 15:43:24.836698 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.836681    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"
Sep 08 15:43:24.836698 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.836691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9" event={"ID":"e46c7178-65f1-43d3-9b80-5dfe2afbafc9","Type":"ContainerDied","Data":"28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7"}
Sep 08 15:43:24.836912 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.836731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9" event={"ID":"e46c7178-65f1-43d3-9b80-5dfe2afbafc9","Type":"ContainerDied","Data":"939ee21c8549c6eb12b0207986cc89c56b826f5e0234d88f13425496aec04fd2"}
Sep 08 15:43:24.836912 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.836755    2575 scope.go:117] "RemoveContainer" containerID="28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7"
Sep 08 15:43:24.846552 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.846533    2575 scope.go:117] "RemoveContainer" containerID="28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7"
Sep 08 15:43:24.846864 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:43:24.846831    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7\": container with ID starting with 28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7 not found: ID does not exist" containerID="28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7"
Sep 08 15:43:24.846929 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.846873    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7"} err="failed to get container status \"28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7\": rpc error: code = NotFound desc = could not find container \"28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7\": container with ID starting with 28e8039a91bfa50f861c47bb6c595ab5479d04336cf9efc20df3f89cb6c6c5f7 not found: ID does not exist"
Sep 08 15:43:24.860823 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.860730    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"]
Sep 08 15:43:24.865165 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.865138    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9fb31f/rate-test-model-9fb31f-kserve-d6fd59fc9-gtpt9"]
Sep 08 15:43:24.900903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:24.900885    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:43:25.441139 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:25.441105    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e46c7178-65f1-43d3-9b80-5dfe2afbafc9" path="/var/lib/kubelet/pods/e46c7178-65f1-43d3-9b80-5dfe2afbafc9/volumes"
Sep 08 15:43:26.800607 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:26.800574    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:43:43.514274 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.514244    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"]
Sep 08 15:43:43.514810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.514590    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e46c7178-65f1-43d3-9b80-5dfe2afbafc9" containerName="main"
Sep 08 15:43:43.514810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.514603    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e46c7178-65f1-43d3-9b80-5dfe2afbafc9" containerName="main"
Sep 08 15:43:43.514810 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.514672    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e46c7178-65f1-43d3-9b80-5dfe2afbafc9" containerName="main"
Sep 08 15:43:43.519009 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.518985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.522100 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.522062    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0fea02-serving-cert\""
Sep 08 15:43:43.529865 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.529784    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"]
Sep 08 15:43:43.609755 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.609731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-metrics-certs\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.609896 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.609787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e9bd9-83f3-473e-a10b-7a1bd9898d53-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.609896 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.609883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55h4l\" (UniqueName: \"kubernetes.io/projected/401e9bd9-83f3-473e-a10b-7a1bd9898d53-kube-api-access-55h4l\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.609978 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.609908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.711166 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.711138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e9bd9-83f3-473e-a10b-7a1bd9898d53-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.711280 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.711187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55h4l\" (UniqueName: \"kubernetes.io/projected/401e9bd9-83f3-473e-a10b-7a1bd9898d53-kube-api-access-55h4l\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.711280 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.711206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.711280 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.711243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-metrics-certs\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.711392 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:43:43.711363    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-serving-cert: secret "maas-api-e2e-ipp-b-0fea02-serving-cert" not found
Sep 08 15:43:43.711444 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:43:43.711432    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls podName:401e9bd9-83f3-473e-a10b-7a1bd9898d53 nodeName:}" failed. No retries permitted until 2026-09-08 15:43:44.211411932 +0000 UTC m=+1573.286704203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls") pod "maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" (UID: "401e9bd9-83f3-473e-a10b-7a1bd9898d53") : secret "maas-api-e2e-ipp-b-0fea02-serving-cert" not found
Sep 08 15:43:43.711801 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.711781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e9bd9-83f3-473e-a10b-7a1bd9898d53-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.713946 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.713924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-metrics-certs\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:43.725856 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:43.725823    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55h4l\" (UniqueName: \"kubernetes.io/projected/401e9bd9-83f3-473e-a10b-7a1bd9898d53-kube-api-access-55h4l\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:44.199342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.199307    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"]
Sep 08 15:43:44.202758 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.202743    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.205913 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.205891    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0fea02\""
Sep 08 15:43:44.206127 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.206110    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0fea02-dockercfg-fkb7t\""
Sep 08 15:43:44.215063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.215043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:44.215921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.215904    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"]
Sep 08 15:43:44.218162 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.218142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:44.316600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.316564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66vf\" (UniqueName: \"kubernetes.io/projected/ff6734d5-c3cb-4672-9835-6886f0e1494f-kube-api-access-g66vf\") pod \"payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.316760 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.316615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6734d5-c3cb-4672-9835-6886f0e1494f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.417933 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.417897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6734d5-c3cb-4672-9835-6886f0e1494f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.418087 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.417983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66vf\" (UniqueName: \"kubernetes.io/projected/ff6734d5-c3cb-4672-9835-6886f0e1494f-kube-api-access-g66vf\") pod \"payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.418551 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.418531    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6734d5-c3cb-4672-9835-6886f0e1494f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.428375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.428350    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66vf\" (UniqueName: \"kubernetes.io/projected/ff6734d5-c3cb-4672-9835-6886f0e1494f-kube-api-access-g66vf\") pod \"payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.432209 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.432179    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:44.512890 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.512862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.704206 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.704183    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"]
Sep 08 15:43:44.706168 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:43:44.706144    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401e9bd9_83f3_473e_a10b_7a1bd9898d53.slice/crio-6c550d2d028ca0396bfdf1b769a72f93bee3990806ddd8f1e21d74d5e4d07a1f WatchSource:0}: Error finding container 6c550d2d028ca0396bfdf1b769a72f93bee3990806ddd8f1e21d74d5e4d07a1f: Status 404 returned error can't find the container with id 6c550d2d028ca0396bfdf1b769a72f93bee3990806ddd8f1e21d74d5e4d07a1f
Sep 08 15:43:44.707478 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.707461    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:43:44.786417 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.786393    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"]
Sep 08 15:43:44.788088 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:43:44.788067    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6734d5_c3cb_4672_9835_6886f0e1494f.slice/crio-1353af89697d91955b8acfd416801b8a699f6088240e5154b63757285eb59bf1 WatchSource:0}: Error finding container 1353af89697d91955b8acfd416801b8a699f6088240e5154b63757285eb59bf1: Status 404 returned error can't find the container with id 1353af89697d91955b8acfd416801b8a699f6088240e5154b63757285eb59bf1
Sep 08 15:43:44.943056 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.943025    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" event={"ID":"401e9bd9-83f3-473e-a10b-7a1bd9898d53","Type":"ContainerStarted","Data":"6c550d2d028ca0396bfdf1b769a72f93bee3990806ddd8f1e21d74d5e4d07a1f"}
Sep 08 15:43:44.944580 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.944555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk" event={"ID":"ff6734d5-c3cb-4672-9835-6886f0e1494f","Type":"ContainerStarted","Data":"a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4"}
Sep 08 15:43:44.944709 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.944605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk" event={"ID":"ff6734d5-c3cb-4672-9835-6886f0e1494f","Type":"ContainerStarted","Data":"1353af89697d91955b8acfd416801b8a699f6088240e5154b63757285eb59bf1"}
Sep 08 15:43:44.944709 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.944692    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:43:44.962950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:44.962900    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk" podStartSLOduration=0.962885657 podStartE2EDuration="962.885657ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:43:44.962027192 +0000 UTC m=+1574.037319508" watchObservedRunningTime="2026-09-08 15:43:44.962885657 +0000 UTC m=+1574.038177947"
Sep 08 15:43:45.950670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:45.950628    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" event={"ID":"401e9bd9-83f3-473e-a10b-7a1bd9898d53","Type":"ContainerStarted","Data":"33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb"}
Sep 08 15:43:45.951099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:45.950875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:45.970222 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:45.970183    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" podStartSLOduration=2.647058122 podStartE2EDuration="2.970170895s" totalImagesPullingTime="323.112773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:43:45.968784951 +0000 UTC m=+1575.044077255" watchObservedRunningTime="2026-09-08 15:43:45.970170895 +0000 UTC m=+1575.045463179"
Sep 08 15:43:48.002099 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:48.002068    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:43:51.960716 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:51.960684    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:43:55.952768 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:43:55.952727    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:44:12.532443 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:12.532406    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk_ff6734d5-c3cb-4672-9835-6886f0e1494f/payload-processing/0.log"
Sep 08 15:44:17.318830 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.318793    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"]
Sep 08 15:44:17.322338 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.322320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.327310 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.327277    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-41de51\"/\"kube-root-ca.crt\""
Sep 08 15:44:17.327431 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.327279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-41de51\"/\"default-dockercfg-5qjrc\""
Sep 08 15:44:17.327431 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.327309    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-41de51\"/\"ipp-route-41de51-kserve-self-signed-certs\""
Sep 08 15:44:17.327431 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.327388    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-41de51\"/\"openshift-service-ca.crt\""
Sep 08 15:44:17.333571 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.333550    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"]
Sep 08 15:44:17.481382 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.481348    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-home\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.481382 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.481381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-dshm\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.481592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.481432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfw5\" (UniqueName: \"kubernetes.io/projected/0020c792-5a6f-487d-adfe-61ecb86ddc96-kube-api-access-tsfw5\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.481592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.481470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-tmp-dir\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.481592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.481546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-model-cache\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.481709 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.481593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0020c792-5a6f-487d-adfe-61ecb86ddc96-tls-certs\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-home\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-dshm\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfw5\" (UniqueName: \"kubernetes.io/projected/0020c792-5a6f-487d-adfe-61ecb86ddc96-kube-api-access-tsfw5\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582739 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-tmp-dir\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582739 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-model-cache\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582739 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0020c792-5a6f-487d-adfe-61ecb86ddc96-tls-certs\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582932 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-home\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.582932 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-tmp-dir\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.583001 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.582980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-model-cache\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.584652 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.584634    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-dshm\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.585001 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.584984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0020c792-5a6f-487d-adfe-61ecb86ddc96-tls-certs\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.591569 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.591545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfw5\" (UniqueName: \"kubernetes.io/projected/0020c792-5a6f-487d-adfe-61ecb86ddc96-kube-api-access-tsfw5\") pod \"ipp-route-41de51-kserve-769db6b6c5-cdwn2\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") " pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.632013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.631988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:17.907765 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:17.907737    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"]
Sep 08 15:44:17.909276 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:44:17.909244    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0020c792_5a6f_487d_adfe_61ecb86ddc96.slice/crio-85a71cf69974f1d22910694c97132a85361999af5593e6ee6bbf09592f41d89d WatchSource:0}: Error finding container 85a71cf69974f1d22910694c97132a85361999af5593e6ee6bbf09592f41d89d: Status 404 returned error can't find the container with id 85a71cf69974f1d22910694c97132a85361999af5593e6ee6bbf09592f41d89d
Sep 08 15:44:18.104958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:18.104922    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2" event={"ID":"0020c792-5a6f-487d-adfe-61ecb86ddc96","Type":"ContainerStarted","Data":"cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144"}
Sep 08 15:44:18.104958 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:18.104962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2" event={"ID":"0020c792-5a6f-487d-adfe-61ecb86ddc96","Type":"ContainerStarted","Data":"85a71cf69974f1d22910694c97132a85361999af5593e6ee6bbf09592f41d89d"}
Sep 08 15:44:18.125211 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:18.125161    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2" podStartSLOduration=1.125147219 podStartE2EDuration="1.125147219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:18.122767417 +0000 UTC m=+1607.198059731" watchObservedRunningTime="2026-09-08 15:44:18.125147219 +0000 UTC m=+1607.200439512"
Sep 08 15:44:23.202304 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:23.202207    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:44:27.632929 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:27.632897    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:27.633387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:27.632949    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:27.645227 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:27.645201    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:28.163901 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:28.163875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:44:43.207826 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:43.207783    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:44:50.667989 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:44:50.667957    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk_ff6734d5-c3cb-4672-9835-6886f0e1494f/payload-processing/0.log"
Sep 08 15:45:00.163335 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.163301    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"]
Sep 08 15:45:00.167102 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.167083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"
Sep 08 15:45:00.170472 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.170450    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9jgf\""
Sep 08 15:45:00.177158 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.177140    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"
Sep 08 15:45:00.184360 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.184338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"]
Sep 08 15:45:00.186697 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.186672    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"]
Sep 08 15:45:00.190602 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.190584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"
Sep 08 15:45:00.200232 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.200204    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"]
Sep 08 15:45:00.216385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.216366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"
Sep 08 15:45:00.464821 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.464788    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"]
Sep 08 15:45:00.467553 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:45:00.467522    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba03a0ee_2f41_47f9_b4ce_fa873a825a02.slice/crio-73f03cfdf270fc7d92856f2dfd753be403e5357274e3ad03109982d65fff8b37 WatchSource:0}: Error finding container 73f03cfdf270fc7d92856f2dfd753be403e5357274e3ad03109982d65fff8b37: Status 404 returned error can't find the container with id 73f03cfdf270fc7d92856f2dfd753be403e5357274e3ad03109982d65fff8b37
Sep 08 15:45:00.499470 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:00.499446    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"]
Sep 08 15:45:00.502148 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:45:00.502127    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9caed809_64c1_4b8e_b97f_e2d05380257d.slice/crio-3e2e5407720d8ef48e5ba74b4d782803db5a34dcc283196a34c2db964c28de4d WatchSource:0}: Error finding container 3e2e5407720d8ef48e5ba74b4d782803db5a34dcc283196a34c2db964c28de4d: Status 404 returned error can't find the container with id 3e2e5407720d8ef48e5ba74b4d782803db5a34dcc283196a34c2db964c28de4d
Sep 08 15:45:01.331604 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.331565    2575 generic.go:356] "Generic (PLEG): container finished" podID="9caed809-64c1-4b8e-b97f-e2d05380257d" containerID="67a93a27cb6114d4ae47d9fd0922e70006a72e2b09b0f4ff7352e12820cf7332" exitCode=0
Sep 08 15:45:01.331987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.331648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl" event={"ID":"9caed809-64c1-4b8e-b97f-e2d05380257d","Type":"ContainerDied","Data":"67a93a27cb6114d4ae47d9fd0922e70006a72e2b09b0f4ff7352e12820cf7332"}
Sep 08 15:45:01.331987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.331713    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl" event={"ID":"9caed809-64c1-4b8e-b97f-e2d05380257d","Type":"ContainerStarted","Data":"3e2e5407720d8ef48e5ba74b4d782803db5a34dcc283196a34c2db964c28de4d"}
Sep 08 15:45:01.333597 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.333576    2575 generic.go:356] "Generic (PLEG): container finished" podID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerID="54cff818a716e20e179e88f45d43d113338a30260787875b5df7651bdc6c4c50" exitCode=7
Sep 08 15:45:01.333688 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.333610    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" event={"ID":"ba03a0ee-2f41-47f9-b4ce-fa873a825a02","Type":"ContainerDied","Data":"54cff818a716e20e179e88f45d43d113338a30260787875b5df7651bdc6c4c50"}
Sep 08 15:45:01.333688 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.333653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" event={"ID":"ba03a0ee-2f41-47f9-b4ce-fa873a825a02","Type":"ContainerStarted","Data":"73f03cfdf270fc7d92856f2dfd753be403e5357274e3ad03109982d65fff8b37"}
Sep 08 15:45:01.333829 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:01.333813    2575 scope.go:117] "RemoveContainer" containerID="54cff818a716e20e179e88f45d43d113338a30260787875b5df7651bdc6c4c50"
Sep 08 15:45:02.340067 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:02.339990    2575 generic.go:356] "Generic (PLEG): container finished" podID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerID="d620b5695d60405c3a1010f45355f7cae72306871db61f3de70fa7593cccdb0f" exitCode=7
Sep 08 15:45:02.340438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:02.340078    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" event={"ID":"ba03a0ee-2f41-47f9-b4ce-fa873a825a02","Type":"ContainerDied","Data":"d620b5695d60405c3a1010f45355f7cae72306871db61f3de70fa7593cccdb0f"}
Sep 08 15:45:02.340438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:02.340139    2575 scope.go:117] "RemoveContainer" containerID="54cff818a716e20e179e88f45d43d113338a30260787875b5df7651bdc6c4c50"
Sep 08 15:45:02.340438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:02.340272    2575 scope.go:117] "RemoveContainer" containerID="d620b5695d60405c3a1010f45355f7cae72306871db61f3de70fa7593cccdb0f"
Sep 08 15:45:02.340616 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:45:02.340510    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb_odh-ai-gateway-infra(ba03a0ee-2f41-47f9-b4ce-fa873a825a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02"
Sep 08 15:45:02.472900 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:02.472872    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"
Sep 08 15:45:03.346653 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:03.346628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl"
Sep 08 15:45:03.347085 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:03.346659    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-tzcxl" event={"ID":"9caed809-64c1-4b8e-b97f-e2d05380257d","Type":"ContainerDied","Data":"3e2e5407720d8ef48e5ba74b4d782803db5a34dcc283196a34c2db964c28de4d"}
Sep 08 15:45:03.347085 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:03.346714    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e2e5407720d8ef48e5ba74b4d782803db5a34dcc283196a34c2db964c28de4d"
Sep 08 15:45:03.348701 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:03.348681    2575 scope.go:117] "RemoveContainer" containerID="d620b5695d60405c3a1010f45355f7cae72306871db61f3de70fa7593cccdb0f"
Sep 08 15:45:03.348900 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:45:03.348881    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb_odh-ai-gateway-infra(ba03a0ee-2f41-47f9-b4ce-fa873a825a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02"
Sep 08 15:45:03.712988 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:03.712950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:45:16.436239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:16.436212    2575 scope.go:117] "RemoveContainer" containerID="d620b5695d60405c3a1010f45355f7cae72306871db61f3de70fa7593cccdb0f"
Sep 08 15:45:17.422619 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:17.422590    2575 generic.go:356] "Generic (PLEG): container finished" podID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerID="64d54260b32392eefc6782a03bcc90e3d00755e95a811c8b8d3bfb92c909dbe8" exitCode=0
Sep 08 15:45:17.422708 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:17.422660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" event={"ID":"ba03a0ee-2f41-47f9-b4ce-fa873a825a02","Type":"ContainerDied","Data":"64d54260b32392eefc6782a03bcc90e3d00755e95a811c8b8d3bfb92c909dbe8"}
Sep 08 15:45:17.422764 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:17.422722    2575 scope.go:117] "RemoveContainer" containerID="d620b5695d60405c3a1010f45355f7cae72306871db61f3de70fa7593cccdb0f"
Sep 08 15:45:18.457360 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:18.457313    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"]
Sep 08 15:45:18.558013 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:18.557991    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"
Sep 08 15:45:19.436348 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:19.436321    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"
Sep 08 15:45:19.441358 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:19.441328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb" event={"ID":"ba03a0ee-2f41-47f9-b4ce-fa873a825a02","Type":"ContainerDied","Data":"73f03cfdf270fc7d92856f2dfd753be403e5357274e3ad03109982d65fff8b37"}
Sep 08 15:45:19.441502 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:19.441371    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f03cfdf270fc7d92856f2dfd753be403e5357274e3ad03109982d65fff8b37"
Sep 08 15:45:19.468439 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:19.468413    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"]
Sep 08 15:45:19.474835 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:19.474814    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-21c704-29814705-bqzrb"]
Sep 08 15:45:21.441538 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:21.441502    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" path="/var/lib/kubelet/pods/ba03a0ee-2f41-47f9-b4ce-fa873a825a02/volumes"
Sep 08 15:45:32.107428 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:32.107385    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:45:53.272236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.272158    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"]
Sep 08 15:45:53.275997 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.272363    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st" podUID="f1b31a9f-bd42-43d5-bebd-02e22ad37d34" containerName="payload-pre-processing" containerID="cri-o://e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054" gracePeriod=30
Sep 08 15:45:53.536624 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.529916    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:45:53.601165 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.601132    2575 generic.go:356] "Generic (PLEG): container finished" podID="f1b31a9f-bd42-43d5-bebd-02e22ad37d34" containerID="e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054" exitCode=0
Sep 08 15:45:53.601307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.601171    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"
Sep 08 15:45:53.601307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.601187    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st" event={"ID":"f1b31a9f-bd42-43d5-bebd-02e22ad37d34","Type":"ContainerDied","Data":"e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054"}
Sep 08 15:45:53.601307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.601230    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st" event={"ID":"f1b31a9f-bd42-43d5-bebd-02e22ad37d34","Type":"ContainerDied","Data":"7bd25efb0b7d645fbcbf6542c334e0c96c890b1abc656faef6ff407a08fd394b"}
Sep 08 15:45:53.601307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.601257    2575 scope.go:117] "RemoveContainer" containerID="e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054"
Sep 08 15:45:53.609695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.609681    2575 scope.go:117] "RemoveContainer" containerID="e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054"
Sep 08 15:45:53.609977 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:45:53.609958    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054\": container with ID starting with e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054 not found: ID does not exist" containerID="e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054"
Sep 08 15:45:53.610033 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.609984    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054"} err="failed to get container status \"e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054\": rpc error: code = NotFound desc = could not find container \"e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054\": container with ID starting with e64205393720d068d9028adb4be056242ddcc1a4fd2846d0e193b6fe2f292054 not found: ID does not exist"
Sep 08 15:45:53.665498 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.665476    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-plugins-config-volume\") pod \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") "
Sep 08 15:45:53.665598 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.665507    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg4zb\" (UniqueName: \"kubernetes.io/projected/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-kube-api-access-jg4zb\") pod \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\" (UID: \"f1b31a9f-bd42-43d5-bebd-02e22ad37d34\") "
Sep 08 15:45:53.665832 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.665810    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1b31a9f-bd42-43d5-bebd-02e22ad37d34" (UID: "f1b31a9f-bd42-43d5-bebd-02e22ad37d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:45:53.667654 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.667629    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-kube-api-access-jg4zb" (OuterVolumeSpecName: "kube-api-access-jg4zb") pod "f1b31a9f-bd42-43d5-bebd-02e22ad37d34" (UID: "f1b31a9f-bd42-43d5-bebd-02e22ad37d34"). InnerVolumeSpecName "kube-api-access-jg4zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:45:53.766345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.766316    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:45:53.766345 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.766341    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg4zb\" (UniqueName: \"kubernetes.io/projected/f1b31a9f-bd42-43d5-bebd-02e22ad37d34-kube-api-access-jg4zb\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:45:53.925137 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.925106    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"]
Sep 08 15:45:53.929034 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:53.929009    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41de51-7bb747454b-v45st"]
Sep 08 15:45:54.297104 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:54.297013    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:45:55.442349 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:55.442311    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1b31a9f-bd42-43d5-bebd-02e22ad37d34" path="/var/lib/kubelet/pods/f1b31a9f-bd42-43d5-bebd-02e22ad37d34/volumes"
Sep 08 15:45:58.799025 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:45:58.798993    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"]
Sep 08 15:46:03.610600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:03.610563    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:46:13.818488 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:13.818424    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2" podUID="0020c792-5a6f-487d-adfe-61ecb86ddc96" containerName="main" containerID="cri-o://cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144" gracePeriod=45
Sep 08 15:46:14.073690 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.073637    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:46:14.236452 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236423    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-tmp-dir\") pod \"0020c792-5a6f-487d-adfe-61ecb86ddc96\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") "
Sep 08 15:46:14.236630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236494    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0020c792-5a6f-487d-adfe-61ecb86ddc96-tls-certs\") pod \"0020c792-5a6f-487d-adfe-61ecb86ddc96\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") "
Sep 08 15:46:14.236630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236514    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-model-cache\") pod \"0020c792-5a6f-487d-adfe-61ecb86ddc96\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") "
Sep 08 15:46:14.236630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236549    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-dshm\") pod \"0020c792-5a6f-487d-adfe-61ecb86ddc96\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") "
Sep 08 15:46:14.236630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236570    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-home\") pod \"0020c792-5a6f-487d-adfe-61ecb86ddc96\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") "
Sep 08 15:46:14.236630 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236591    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsfw5\" (UniqueName: \"kubernetes.io/projected/0020c792-5a6f-487d-adfe-61ecb86ddc96-kube-api-access-tsfw5\") pod \"0020c792-5a6f-487d-adfe-61ecb86ddc96\" (UID: \"0020c792-5a6f-487d-adfe-61ecb86ddc96\") "
Sep 08 15:46:14.236934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236720    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0020c792-5a6f-487d-adfe-61ecb86ddc96" (UID: "0020c792-5a6f-487d-adfe-61ecb86ddc96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:46:14.236934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236821    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-home" (OuterVolumeSpecName: "home") pod "0020c792-5a6f-487d-adfe-61ecb86ddc96" (UID: "0020c792-5a6f-487d-adfe-61ecb86ddc96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:46:14.236934 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236839    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-model-cache" (OuterVolumeSpecName: "model-cache") pod "0020c792-5a6f-487d-adfe-61ecb86ddc96" (UID: "0020c792-5a6f-487d-adfe-61ecb86ddc96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:46:14.237083 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.236944    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:14.239006 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.238987    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0020c792-5a6f-487d-adfe-61ecb86ddc96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0020c792-5a6f-487d-adfe-61ecb86ddc96" (UID: "0020c792-5a6f-487d-adfe-61ecb86ddc96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:14.239091 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.238988    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-dshm" (OuterVolumeSpecName: "dshm") pod "0020c792-5a6f-487d-adfe-61ecb86ddc96" (UID: "0020c792-5a6f-487d-adfe-61ecb86ddc96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:46:14.239091 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.239072    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0020c792-5a6f-487d-adfe-61ecb86ddc96-kube-api-access-tsfw5" (OuterVolumeSpecName: "kube-api-access-tsfw5") pod "0020c792-5a6f-487d-adfe-61ecb86ddc96" (UID: "0020c792-5a6f-487d-adfe-61ecb86ddc96"). InnerVolumeSpecName "kube-api-access-tsfw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:14.337684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.337631    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:14.337684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.337652    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfw5\" (UniqueName: \"kubernetes.io/projected/0020c792-5a6f-487d-adfe-61ecb86ddc96-kube-api-access-tsfw5\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:14.337684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.337664    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0020c792-5a6f-487d-adfe-61ecb86ddc96-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:14.337684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.337673    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:14.337684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.337681    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0020c792-5a6f-487d-adfe-61ecb86ddc96-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:14.712409 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.712369    2575 generic.go:356] "Generic (PLEG): container finished" podID="0020c792-5a6f-487d-adfe-61ecb86ddc96" containerID="cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144" exitCode=0
Sep 08 15:46:14.712567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.712416    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"
Sep 08 15:46:14.712567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.712414    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2" event={"ID":"0020c792-5a6f-487d-adfe-61ecb86ddc96","Type":"ContainerDied","Data":"cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144"}
Sep 08 15:46:14.712567 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.712540    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2" event={"ID":"0020c792-5a6f-487d-adfe-61ecb86ddc96","Type":"ContainerDied","Data":"85a71cf69974f1d22910694c97132a85361999af5593e6ee6bbf09592f41d89d"}
Sep 08 15:46:14.712680 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.712567    2575 scope.go:117] "RemoveContainer" containerID="cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144"
Sep 08 15:46:14.721972 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.721920    2575 scope.go:117] "RemoveContainer" containerID="cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144"
Sep 08 15:46:14.722230 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:46:14.722211    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144\": container with ID starting with cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144 not found: ID does not exist" containerID="cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144"
Sep 08 15:46:14.722274 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.722237    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144"} err="failed to get container status \"cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144\": rpc error: code = NotFound desc = could not find container \"cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144\": container with ID starting with cfa264a65ef7e2a05893918dd1491ab33e1633e7f3b834178335d471d8de0144 not found: ID does not exist"
Sep 08 15:46:14.736023 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.735999    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"]
Sep 08 15:46:14.741133 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:14.741114    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-41de51/ipp-route-41de51-kserve-769db6b6c5-cdwn2"]
Sep 08 15:46:15.442371 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:15.442337    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0020c792-5a6f-487d-adfe-61ecb86ddc96" path="/var/lib/kubelet/pods/0020c792-5a6f-487d-adfe-61ecb86ddc96/volumes"
Sep 08 15:46:53.221835 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.221803    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"]
Sep 08 15:46:53.222393 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.222027    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" podUID="401e9bd9-83f3-473e-a10b-7a1bd9898d53" containerName="maas-api" containerID="cri-o://33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb" gracePeriod=30
Sep 08 15:46:53.291086 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.291058    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"]
Sep 08 15:46:53.291341 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.291316    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk" podUID="ff6734d5-c3cb-4672-9835-6886f0e1494f" containerName="payload-processing" containerID="cri-o://a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4" gracePeriod=30
Sep 08 15:46:53.526828 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.526803    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:46:53.570518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.570486    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:46:53.626974 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.626950    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-metrics-certs\") pod \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") "
Sep 08 15:46:53.627140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627013    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55h4l\" (UniqueName: \"kubernetes.io/projected/401e9bd9-83f3-473e-a10b-7a1bd9898d53-kube-api-access-55h4l\") pod \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") "
Sep 08 15:46:53.627140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627031    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g66vf\" (UniqueName: \"kubernetes.io/projected/ff6734d5-c3cb-4672-9835-6886f0e1494f-kube-api-access-g66vf\") pod \"ff6734d5-c3cb-4672-9835-6886f0e1494f\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") "
Sep 08 15:46:53.627140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627049    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6734d5-c3cb-4672-9835-6886f0e1494f-plugins-config-volume\") pod \"ff6734d5-c3cb-4672-9835-6886f0e1494f\" (UID: \"ff6734d5-c3cb-4672-9835-6886f0e1494f\") "
Sep 08 15:46:53.627140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627067    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e9bd9-83f3-473e-a10b-7a1bd9898d53-openshift-service-ca\") pod \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") "
Sep 08 15:46:53.627140 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627094    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls\") pod \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\" (UID: \"401e9bd9-83f3-473e-a10b-7a1bd9898d53\") "
Sep 08 15:46:53.627504 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627476    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6734d5-c3cb-4672-9835-6886f0e1494f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff6734d5-c3cb-4672-9835-6886f0e1494f" (UID: "ff6734d5-c3cb-4672-9835-6886f0e1494f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:53.627504 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.627479    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401e9bd9-83f3-473e-a10b-7a1bd9898d53-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "401e9bd9-83f3-473e-a10b-7a1bd9898d53" (UID: "401e9bd9-83f3-473e-a10b-7a1bd9898d53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:53.629307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.629283    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "401e9bd9-83f3-473e-a10b-7a1bd9898d53" (UID: "401e9bd9-83f3-473e-a10b-7a1bd9898d53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:53.629307 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.629294    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6734d5-c3cb-4672-9835-6886f0e1494f-kube-api-access-g66vf" (OuterVolumeSpecName: "kube-api-access-g66vf") pod "ff6734d5-c3cb-4672-9835-6886f0e1494f" (UID: "ff6734d5-c3cb-4672-9835-6886f0e1494f"). InnerVolumeSpecName "kube-api-access-g66vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:53.629431 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.629312    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401e9bd9-83f3-473e-a10b-7a1bd9898d53-kube-api-access-55h4l" (OuterVolumeSpecName: "kube-api-access-55h4l") pod "401e9bd9-83f3-473e-a10b-7a1bd9898d53" (UID: "401e9bd9-83f3-473e-a10b-7a1bd9898d53"). InnerVolumeSpecName "kube-api-access-55h4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:53.629529 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.629512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "401e9bd9-83f3-473e-a10b-7a1bd9898d53" (UID: "401e9bd9-83f3-473e-a10b-7a1bd9898d53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:53.728239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.728185    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:53.728239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.728206    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55h4l\" (UniqueName: \"kubernetes.io/projected/401e9bd9-83f3-473e-a10b-7a1bd9898d53-kube-api-access-55h4l\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:53.728239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.728216    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66vf\" (UniqueName: \"kubernetes.io/projected/ff6734d5-c3cb-4672-9835-6886f0e1494f-kube-api-access-g66vf\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:53.728239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.728225    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6734d5-c3cb-4672-9835-6886f0e1494f-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:53.728239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.728233    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e9bd9-83f3-473e-a10b-7a1bd9898d53-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:53.728239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.728241    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401e9bd9-83f3-473e-a10b-7a1bd9898d53-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:46:53.909947 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.909921    2575 generic.go:356] "Generic (PLEG): container finished" podID="ff6734d5-c3cb-4672-9835-6886f0e1494f" containerID="a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4" exitCode=0
Sep 08 15:46:53.910041 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.909966    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"
Sep 08 15:46:53.910041 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.910007    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk" event={"ID":"ff6734d5-c3cb-4672-9835-6886f0e1494f","Type":"ContainerDied","Data":"a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4"}
Sep 08 15:46:53.910136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.910099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk" event={"ID":"ff6734d5-c3cb-4672-9835-6886f0e1494f","Type":"ContainerDied","Data":"1353af89697d91955b8acfd416801b8a699f6088240e5154b63757285eb59bf1"}
Sep 08 15:46:53.910136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.910127    2575 scope.go:117] "RemoveContainer" containerID="a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4"
Sep 08 15:46:53.911505 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.911479    2575 generic.go:356] "Generic (PLEG): container finished" podID="401e9bd9-83f3-473e-a10b-7a1bd9898d53" containerID="33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb" exitCode=0
Sep 08 15:46:53.911619 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.911510    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" event={"ID":"401e9bd9-83f3-473e-a10b-7a1bd9898d53","Type":"ContainerDied","Data":"33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb"}
Sep 08 15:46:53.911619 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.911545    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"
Sep 08 15:46:53.911793 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.911555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw" event={"ID":"401e9bd9-83f3-473e-a10b-7a1bd9898d53","Type":"ContainerDied","Data":"6c550d2d028ca0396bfdf1b769a72f93bee3990806ddd8f1e21d74d5e4d07a1f"}
Sep 08 15:46:53.919202 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.919181    2575 scope.go:117] "RemoveContainer" containerID="a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4"
Sep 08 15:46:53.919572 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:46:53.919469    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4\": container with ID starting with a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4 not found: ID does not exist" containerID="a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4"
Sep 08 15:46:53.919572 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.919492    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4"} err="failed to get container status \"a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4\": rpc error: code = NotFound desc = could not find container \"a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4\": container with ID starting with a97823699e9945806031cba0003930113da1acb7456b6d9591fdb011795d8cd4 not found: ID does not exist"
Sep 08 15:46:53.919572 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.919513    2575 scope.go:117] "RemoveContainer" containerID="33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb"
Sep 08 15:46:53.927999 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.927982    2575 scope.go:117] "RemoveContainer" containerID="33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb"
Sep 08 15:46:53.928273 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:46:53.928255    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb\": container with ID starting with 33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb not found: ID does not exist" containerID="33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb"
Sep 08 15:46:53.928314 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.928282    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb"} err="failed to get container status \"33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb\": rpc error: code = NotFound desc = could not find container \"33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb\": container with ID starting with 33bdd891384e6fd5db3598afcb766863f4a4ebef8e2d9b6bfe9fd957ed78a2bb not found: ID does not exist"
Sep 08 15:46:53.936632 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.936611    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"]
Sep 08 15:46:53.941168 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.941145    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0fea02-d88995c6f-7cnsw"]
Sep 08 15:46:53.954565 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.954545    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"]
Sep 08 15:46:53.957604 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:53.957587    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0fea02-65589b7d8c-xtjmk"]
Sep 08 15:46:55.441782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:55.441740    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401e9bd9-83f3-473e-a10b-7a1bd9898d53" path="/var/lib/kubelet/pods/401e9bd9-83f3-473e-a10b-7a1bd9898d53/volumes"
Sep 08 15:46:55.442243 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:55.442140    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6734d5-c3cb-4672-9835-6886f0e1494f" path="/var/lib/kubelet/pods/ff6734d5-c3cb-4672-9835-6886f0e1494f/volumes"
Sep 08 15:46:56.522276 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:46:56.522230    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:47:11.777184 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:11.777142    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"]
Sep 08 15:47:11.777660 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:11.777358    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf" podUID="8aea44b8-26dc-41bd-9981-ce793c931939" containerName="payload-processing" containerID="cri-o://433ba5a84ba7b5e602d4bad1337e1cdf5c656337a438d09c94330c65e565a983" gracePeriod=30
Sep 08 15:47:12.013163 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.013128    2575 generic.go:356] "Generic (PLEG): container finished" podID="8aea44b8-26dc-41bd-9981-ce793c931939" containerID="433ba5a84ba7b5e602d4bad1337e1cdf5c656337a438d09c94330c65e565a983" exitCode=0
Sep 08 15:47:12.013298 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.013189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf" event={"ID":"8aea44b8-26dc-41bd-9981-ce793c931939","Type":"ContainerDied","Data":"433ba5a84ba7b5e602d4bad1337e1cdf5c656337a438d09c94330c65e565a983"}
Sep 08 15:47:12.025413 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.025387    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:47:12.175194 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.175167    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwgmv\" (UniqueName: \"kubernetes.io/projected/8aea44b8-26dc-41bd-9981-ce793c931939-kube-api-access-rwgmv\") pod \"8aea44b8-26dc-41bd-9981-ce793c931939\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") "
Sep 08 15:47:12.175337 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.175237    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aea44b8-26dc-41bd-9981-ce793c931939-plugins-config-volume\") pod \"8aea44b8-26dc-41bd-9981-ce793c931939\" (UID: \"8aea44b8-26dc-41bd-9981-ce793c931939\") "
Sep 08 15:47:12.175604 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.175580    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aea44b8-26dc-41bd-9981-ce793c931939-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8aea44b8-26dc-41bd-9981-ce793c931939" (UID: "8aea44b8-26dc-41bd-9981-ce793c931939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:12.177376 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.177351    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aea44b8-26dc-41bd-9981-ce793c931939-kube-api-access-rwgmv" (OuterVolumeSpecName: "kube-api-access-rwgmv") pod "8aea44b8-26dc-41bd-9981-ce793c931939" (UID: "8aea44b8-26dc-41bd-9981-ce793c931939"). InnerVolumeSpecName "kube-api-access-rwgmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:12.276385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.276359    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aea44b8-26dc-41bd-9981-ce793c931939-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:12.276385 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:12.276382    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwgmv\" (UniqueName: \"kubernetes.io/projected/8aea44b8-26dc-41bd-9981-ce793c931939-kube-api-access-rwgmv\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:13.020154 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:13.020124    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"
Sep 08 15:47:13.020154 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:13.020139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf" event={"ID":"8aea44b8-26dc-41bd-9981-ce793c931939","Type":"ContainerDied","Data":"6b3e52a59d8040cbb0ed88f29772b3c5690bc3574c3de6a787bfea3cf56ba050"}
Sep 08 15:47:13.020612 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:13.020210    2575 scope.go:117] "RemoveContainer" containerID="433ba5a84ba7b5e602d4bad1337e1cdf5c656337a438d09c94330c65e565a983"
Sep 08 15:47:13.043136 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:13.043109    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"]
Sep 08 15:47:13.046001 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:13.045974    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ca1fc-765b6ffdfc-m4nrf"]
Sep 08 15:47:13.441601 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:13.441560    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aea44b8-26dc-41bd-9981-ce793c931939" path="/var/lib/kubelet/pods/8aea44b8-26dc-41bd-9981-ce793c931939/volumes"
Sep 08 15:47:14.503525 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:14.503498    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:47:19.429012 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:19.428980    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"]
Sep 08 15:47:19.429387 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:19.429215    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy" containerID="cri-o://2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2" gracePeriod=30
Sep 08 15:47:24.674809 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.674787    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:47:24.771774 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771713    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-certs\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.771774 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771755    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-socket\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.771774 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771771    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-envoy\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771789    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-data\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771806    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-podinfo\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771828    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-credential-socket\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771888    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nhzr\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-kube-api-access-9nhzr\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-token\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772036 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.771988    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c443c10-40ed-4406-9c12-f550fd02bad7-istiod-ca-cert\") pod \"7c443c10-40ed-4406-9c12-f550fd02bad7\" (UID: \"7c443c10-40ed-4406-9c12-f550fd02bad7\") "
Sep 08 15:47:24.772330 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772040    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:24.772330 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772087    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:24.772330 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772101    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:24.772330 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772230    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:24.772330 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772305    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-data\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.772330 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772323    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.772827 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772337    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-workload-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.772827 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.772386    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c443c10-40ed-4406-9c12-f550fd02bad7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:24.774192 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.774158    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:24.774302 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.774216    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:47:24.774365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.774335    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:24.774365 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.774351    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-kube-api-access-9nhzr" (OuterVolumeSpecName: "kube-api-access-9nhzr") pod "7c443c10-40ed-4406-9c12-f550fd02bad7" (UID: "7c443c10-40ed-4406-9c12-f550fd02bad7"). InnerVolumeSpecName "kube-api-access-9nhzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:24.873204 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.873182    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.873204 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.873201    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c443c10-40ed-4406-9c12-f550fd02bad7-istiod-ca-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.873338 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.873211    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-envoy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.873338 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.873218    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c443c10-40ed-4406-9c12-f550fd02bad7-istio-podinfo\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.873338 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.873226    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c443c10-40ed-4406-9c12-f550fd02bad7-credential-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:24.873338 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:24.873234    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhzr\" (UniqueName: \"kubernetes.io/projected/7c443c10-40ed-4406-9c12-f550fd02bad7-kube-api-access-9nhzr\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:25.080127 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.080054    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerID="2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2" exitCode=0
Sep 08 15:47:25.080259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.080130    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"
Sep 08 15:47:25.080259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.080143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" event={"ID":"7c443c10-40ed-4406-9c12-f550fd02bad7","Type":"ContainerDied","Data":"2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2"}
Sep 08 15:47:25.080259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.080225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z" event={"ID":"7c443c10-40ed-4406-9c12-f550fd02bad7","Type":"ContainerDied","Data":"0228a40c858487cfb7eb3de6a8e238b330c502b97085c879ed9274314f992afc"}
Sep 08 15:47:25.080259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.080253    2575 scope.go:117] "RemoveContainer" containerID="2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2"
Sep 08 15:47:25.090333 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.090314    2575 scope.go:117] "RemoveContainer" containerID="d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"
Sep 08 15:47:25.098837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.098822    2575 scope.go:117] "RemoveContainer" containerID="2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2"
Sep 08 15:47:25.099123 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:47:25.099104    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2\": container with ID starting with 2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2 not found: ID does not exist" containerID="2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2"
Sep 08 15:47:25.099168 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.099132    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2"} err="failed to get container status \"2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2\": rpc error: code = NotFound desc = could not find container \"2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2\": container with ID starting with 2a5d7d3b304edaa63a5888d659b7d7ec0c1ad22024ab927c91e086f7d9a3fba2 not found: ID does not exist"
Sep 08 15:47:25.099168 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.099151    2575 scope.go:117] "RemoveContainer" containerID="d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"
Sep 08 15:47:25.099433 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:47:25.099415    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255\": container with ID starting with d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255 not found: ID does not exist" containerID="d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"
Sep 08 15:47:25.099480 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.099438    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255"} err="failed to get container status \"d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255\": rpc error: code = NotFound desc = could not find container \"d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255\": container with ID starting with d1086b6a0c59ba49b5083c442325a5cda0b4653813859f10f9b5b7c8a5c27255 not found: ID does not exist"
Sep 08 15:47:25.104895 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.104871    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"]
Sep 08 15:47:25.108776 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.108750    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ca1fc-openshift-default-688754cc74-7nh7z"]
Sep 08 15:47:25.441977 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:25.441949    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" path="/var/lib/kubelet/pods/7c443c10-40ed-4406-9c12-f550fd02bad7/volumes"
Sep 08 15:47:30.504685 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.504649    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"]
Sep 08 15:47:30.505209 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.504885    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d" podUID="2985b6fb-34d3-4093-99a1-14fbd57d41af" containerName="maas-api" containerID="cri-o://83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd" gracePeriod=30
Sep 08 15:47:30.586742 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.586705    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"]
Sep 08 15:47:30.586948 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.586924    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd" podUID="d52faf85-1168-4a5d-8b67-a49df1f373ce" containerName="payload-pre-processing" containerID="cri-o://eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319" gracePeriod=30
Sep 08 15:47:30.819477 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.819454    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:47:30.851743 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.851722    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:47:30.926940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.921201    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985b6fb-34d3-4093-99a1-14fbd57d41af-openshift-service-ca\") pod \"2985b6fb-34d3-4093-99a1-14fbd57d41af\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") "
Sep 08 15:47:30.926940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.921251    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-metrics-certs\") pod \"2985b6fb-34d3-4093-99a1-14fbd57d41af\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") "
Sep 08 15:47:30.926940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.921340    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8r88\" (UniqueName: \"kubernetes.io/projected/2985b6fb-34d3-4093-99a1-14fbd57d41af-kube-api-access-k8r88\") pod \"2985b6fb-34d3-4093-99a1-14fbd57d41af\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") "
Sep 08 15:47:30.926940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.921382    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-maas-api-tls\") pod \"2985b6fb-34d3-4093-99a1-14fbd57d41af\" (UID: \"2985b6fb-34d3-4093-99a1-14fbd57d41af\") "
Sep 08 15:47:30.926940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.924795    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2985b6fb-34d3-4093-99a1-14fbd57d41af" (UID: "2985b6fb-34d3-4093-99a1-14fbd57d41af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:30.926940 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.925153    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2985b6fb-34d3-4093-99a1-14fbd57d41af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2985b6fb-34d3-4093-99a1-14fbd57d41af" (UID: "2985b6fb-34d3-4093-99a1-14fbd57d41af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:30.927443 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.927421    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2985b6fb-34d3-4093-99a1-14fbd57d41af" (UID: "2985b6fb-34d3-4093-99a1-14fbd57d41af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:30.928739 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:30.928716    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2985b6fb-34d3-4093-99a1-14fbd57d41af-kube-api-access-k8r88" (OuterVolumeSpecName: "kube-api-access-k8r88") pod "2985b6fb-34d3-4093-99a1-14fbd57d41af" (UID: "2985b6fb-34d3-4093-99a1-14fbd57d41af"). InnerVolumeSpecName "kube-api-access-k8r88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:31.022100 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022079    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52faf85-1168-4a5d-8b67-a49df1f373ce-plugins-config-volume\") pod \"d52faf85-1168-4a5d-8b67-a49df1f373ce\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") "
Sep 08 15:47:31.022201 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022121    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4szwk\" (UniqueName: \"kubernetes.io/projected/d52faf85-1168-4a5d-8b67-a49df1f373ce-kube-api-access-4szwk\") pod \"d52faf85-1168-4a5d-8b67-a49df1f373ce\" (UID: \"d52faf85-1168-4a5d-8b67-a49df1f373ce\") "
Sep 08 15:47:31.022267 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022253    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8r88\" (UniqueName: \"kubernetes.io/projected/2985b6fb-34d3-4093-99a1-14fbd57d41af-kube-api-access-k8r88\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:31.022312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022271    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:31.022312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022281    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2985b6fb-34d3-4093-99a1-14fbd57d41af-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:31.022312 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022289    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2985b6fb-34d3-4093-99a1-14fbd57d41af-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:31.022410 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.022392    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52faf85-1168-4a5d-8b67-a49df1f373ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d52faf85-1168-4a5d-8b67-a49df1f373ce" (UID: "d52faf85-1168-4a5d-8b67-a49df1f373ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:31.024149 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.024131    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52faf85-1168-4a5d-8b67-a49df1f373ce-kube-api-access-4szwk" (OuterVolumeSpecName: "kube-api-access-4szwk") pod "d52faf85-1168-4a5d-8b67-a49df1f373ce" (UID: "d52faf85-1168-4a5d-8b67-a49df1f373ce"). InnerVolumeSpecName "kube-api-access-4szwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:31.109394 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.109334    2575 generic.go:356] "Generic (PLEG): container finished" podID="2985b6fb-34d3-4093-99a1-14fbd57d41af" containerID="83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd" exitCode=0
Sep 08 15:47:31.109481 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.109392    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"
Sep 08 15:47:31.109481 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.109423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d" event={"ID":"2985b6fb-34d3-4093-99a1-14fbd57d41af","Type":"ContainerDied","Data":"83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd"}
Sep 08 15:47:31.109584 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.109506    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d" event={"ID":"2985b6fb-34d3-4093-99a1-14fbd57d41af","Type":"ContainerDied","Data":"0089160b56554c92f6c282814004ef5511af0954968599a05c3af9eb143af5ec"}
Sep 08 15:47:31.109584 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.109534    2575 scope.go:117] "RemoveContainer" containerID="83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd"
Sep 08 15:47:31.110921 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.110898    2575 generic.go:356] "Generic (PLEG): container finished" podID="d52faf85-1168-4a5d-8b67-a49df1f373ce" containerID="eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319" exitCode=0
Sep 08 15:47:31.111002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.110926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd" event={"ID":"d52faf85-1168-4a5d-8b67-a49df1f373ce","Type":"ContainerDied","Data":"eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319"}
Sep 08 15:47:31.111002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.110955    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"
Sep 08 15:47:31.111002 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.110970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd" event={"ID":"d52faf85-1168-4a5d-8b67-a49df1f373ce","Type":"ContainerDied","Data":"fa22da4623c4da868f605e63a651e4e87f75295463b62231baaaffe56e9d4161"}
Sep 08 15:47:31.120375 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.120360    2575 scope.go:117] "RemoveContainer" containerID="83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd"
Sep 08 15:47:31.120619 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:47:31.120599    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd\": container with ID starting with 83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd not found: ID does not exist" containerID="83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd"
Sep 08 15:47:31.120674 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.120625    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd"} err="failed to get container status \"83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd\": rpc error: code = NotFound desc = could not find container \"83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd\": container with ID starting with 83c8cecbb3d7fb81607d6a9eb73c8932bd56a7d85a34fe37cf6369fbf55abafd not found: ID does not exist"
Sep 08 15:47:31.120674 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.120641    2575 scope.go:117] "RemoveContainer" containerID="eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319"
Sep 08 15:47:31.122713 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.122692    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52faf85-1168-4a5d-8b67-a49df1f373ce-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:31.122784 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.122720    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szwk\" (UniqueName: \"kubernetes.io/projected/d52faf85-1168-4a5d-8b67-a49df1f373ce-kube-api-access-4szwk\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:47:31.131392 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.131373    2575 scope.go:117] "RemoveContainer" containerID="eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319"
Sep 08 15:47:31.131658 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:47:31.131639    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319\": container with ID starting with eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319 not found: ID does not exist" containerID="eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319"
Sep 08 15:47:31.131714 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.131664    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319"} err="failed to get container status \"eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319\": rpc error: code = NotFound desc = could not find container \"eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319\": container with ID starting with eddb0d6ee4338842947d641666ba258be8d147f338c005ea8223a31edb96c319 not found: ID does not exist"
Sep 08 15:47:31.140743 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.140719    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"]
Sep 08 15:47:31.146563 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.146545    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9fb31f-754f6485d-j2lsd"]
Sep 08 15:47:31.160108 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.160089    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"]
Sep 08 15:47:31.163058 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.163039    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9fb31f-7bbc48765c-ltj4d"]
Sep 08 15:47:31.440968 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.440945    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2985b6fb-34d3-4093-99a1-14fbd57d41af" path="/var/lib/kubelet/pods/2985b6fb-34d3-4093-99a1-14fbd57d41af/volumes"
Sep 08 15:47:31.441394 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:31.441371    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52faf85-1168-4a5d-8b67-a49df1f373ce" path="/var/lib/kubelet/pods/d52faf85-1168-4a5d-8b67-a49df1f373ce/volumes"
Sep 08 15:47:33.310862 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:33.310824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:47:41.315868 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:47:41.315814    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:49:02.731534 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:49:02.731441    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:49:26.903615 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:49:26.903577    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:49:34.697527 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:49:34.697480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:49:41.692531 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:49:41.692495    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:49:49.399581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:49:49.399547    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:50:06.793327 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:50:06.793286    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:50:15.200437 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:50:15.200401    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:51:31.702120 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:31.702082    2575 scope.go:117] "RemoveContainer" containerID="64d54260b32392eefc6782a03bcc90e3d00755e95a811c8b8d3bfb92c909dbe8"
Sep 08 15:51:42.394466 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:42.394423    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:51:49.599329 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:49.599286    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:51:57.296584 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:57.296486    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:51:58.204529 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204495    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"]
Sep 08 15:51:58.204830 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204817    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985b6fb-34d3-4093-99a1-14fbd57d41af" containerName="maas-api"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204832    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985b6fb-34d3-4093-99a1-14fbd57d41af" containerName="maas-api"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204861    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6734d5-c3cb-4672-9835-6886f0e1494f" containerName="payload-processing"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204871    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6734d5-c3cb-4672-9835-6886f0e1494f" containerName="payload-processing"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204880    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401e9bd9-83f3-473e-a10b-7a1bd9898d53" containerName="maas-api"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204885    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="401e9bd9-83f3-473e-a10b-7a1bd9898d53" containerName="maas-api"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204891    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aea44b8-26dc-41bd-9981-ce793c931939" containerName="payload-processing"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204898    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aea44b8-26dc-41bd-9981-ce793c931939" containerName="payload-processing"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204905    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9caed809-64c1-4b8e-b97f-e2d05380257d" containerName="cleanup"
Sep 08 15:51:58.204909 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204910    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9caed809-64c1-4b8e-b97f-e2d05380257d" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204917    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204923    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204932    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204937    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204948    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52faf85-1168-4a5d-8b67-a49df1f373ce" containerName="payload-pre-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204953    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52faf85-1168-4a5d-8b67-a49df1f373ce" containerName="payload-pre-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204959    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204964    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204969    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204974    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204980    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1b31a9f-bd42-43d5-bebd-02e22ad37d34" containerName="payload-pre-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204986    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1b31a9f-bd42-43d5-bebd-02e22ad37d34" containerName="payload-pre-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204993    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0020c792-5a6f-487d-adfe-61ecb86ddc96" containerName="main"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.204999    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0020c792-5a6f-487d-adfe-61ecb86ddc96" containerName="main"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205050    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205059    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1b31a9f-bd42-43d5-bebd-02e22ad37d34" containerName="payload-pre-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205064    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205070    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2985b6fb-34d3-4093-99a1-14fbd57d41af" containerName="maas-api"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205075    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9caed809-64c1-4b8e-b97f-e2d05380257d" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205079    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205084    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52faf85-1168-4a5d-8b67-a49df1f373ce" containerName="payload-pre-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205089    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6734d5-c3cb-4672-9835-6886f0e1494f" containerName="payload-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205094    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aea44b8-26dc-41bd-9981-ce793c931939" containerName="payload-processing"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205100    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="401e9bd9-83f3-473e-a10b-7a1bd9898d53" containerName="maas-api"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205104    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0020c792-5a6f-487d-adfe-61ecb86ddc96" containerName="main"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205111    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy"
Sep 08 15:51:58.205213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.205116    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c443c10-40ed-4406-9c12-f550fd02bad7" containerName="istio-proxy"
Sep 08 15:51:58.211557 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.209323    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.213163 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.213138    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bc1e7aad-kserve-self-signed-certs\""
Sep 08 15:51:58.220535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.220507    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"]
Sep 08 15:51:58.281977 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.281951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-dshm\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.282100 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.281998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-model-cache\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.282100 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.282016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tls-certs\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.282175 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.282092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-home\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.282175 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.282141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tmp-dir\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.282175 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.282170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkr8l\" (UniqueName: \"kubernetes.io/projected/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-kube-api-access-lkr8l\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.382815 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.382784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tmp-dir\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.382823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkr8l\" (UniqueName: \"kubernetes.io/projected/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-kube-api-access-lkr8l\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.382909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-dshm\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.382970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-model-cache\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.382998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tls-certs\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.383045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-home\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383242 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.383175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tmp-dir\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383533 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.383293    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-model-cache\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.383533 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.383396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-home\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.385227 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.385204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-dshm\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.385535 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.385520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tls-certs\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.391141 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.391118    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkr8l\" (UniqueName: \"kubernetes.io/projected/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-kube-api-access-lkr8l\") pod \"e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") " pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.522911 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.522837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:51:58.793746 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:58.793720    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"]
Sep 08 15:51:58.795926 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:51:58.795903    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d1c2bbd_56e8_4d1f_bbec_3eff337c6de8.slice/crio-db851a2ac67dfaeb9830027eb5c06ab91d57bcfa6589831e33880c8073f92740 WatchSource:0}: Error finding container db851a2ac67dfaeb9830027eb5c06ab91d57bcfa6589831e33880c8073f92740: Status 404 returned error can't find the container with id db851a2ac67dfaeb9830027eb5c06ab91d57bcfa6589831e33880c8073f92740
Sep 08 15:51:59.351913 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:59.351865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh" event={"ID":"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8","Type":"ContainerStarted","Data":"a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c"}
Sep 08 15:51:59.352102 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:51:59.351924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh" event={"ID":"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8","Type":"ContainerStarted","Data":"db851a2ac67dfaeb9830027eb5c06ab91d57bcfa6589831e33880c8073f92740"}
Sep 08 15:52:04.390144 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:04.390084    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh" podStartSLOduration=6.390062177 podStartE2EDuration="6.390062177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:59.370449524 +0000 UTC m=+2068.445741838" watchObservedRunningTime="2026-09-08 15:52:04.390062177 +0000 UTC m=+2073.465354471"
Sep 08 15:52:04.391695 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:04.391671    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:52:08.523338 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:08.523306    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:52:08.523711 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:08.523352    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:52:08.536172 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:08.536146    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:52:09.410976 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:09.410951    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:52:25.403789 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.403747    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"]
Sep 08 15:52:25.404306 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.404284    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:52:25.404354 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.404312    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba03a0ee-2f41-47f9-b4ce-fa873a825a02" containerName="cleanup"
Sep 08 15:52:25.407782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.407760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.410679 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.410656    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bc1e7aad-kserve-self-signed-certs\""
Sep 08 15:52:25.416276 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.416180    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"]
Sep 08 15:52:25.494705 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.494670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-dshm\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.494881 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.494713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-tmp-dir\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.494881 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.494796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-model-cache\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.494881 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.494827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8km\" (UniqueName: \"kubernetes.io/projected/ea2d196e-d319-4593-8139-4651c5493784-kube-api-access-wd8km\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.495063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.494916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-home\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.495063 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.494940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2d196e-d319-4593-8139-4651c5493784-tls-certs\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596283 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-model-cache\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8km\" (UniqueName: \"kubernetes.io/projected/ea2d196e-d319-4593-8139-4651c5493784-kube-api-access-wd8km\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-home\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2d196e-d319-4593-8139-4651c5493784-tls-certs\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-dshm\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-tmp-dir\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596745 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-model-cache\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596751    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-home\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.596803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.596762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-tmp-dir\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.598787 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.598763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-dshm\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.598899 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.598818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2d196e-d319-4593-8139-4651c5493784-tls-certs\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.605588 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.605564    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8km\" (UniqueName: \"kubernetes.io/projected/ea2d196e-d319-4593-8139-4651c5493784-kube-api-access-wd8km\") pod \"e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") " pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.721373 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.721315    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:25.993684 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:25.993657    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"]
Sep 08 15:52:25.996287 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:52:25.996257    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea2d196e_d319_4593_8139_4651c5493784.slice/crio-126fcaaf1440ad15c459c9823d17d4ea4340b6257a76685e74176c51dad1b96c WatchSource:0}: Error finding container 126fcaaf1440ad15c459c9823d17d4ea4340b6257a76685e74176c51dad1b96c: Status 404 returned error can't find the container with id 126fcaaf1440ad15c459c9823d17d4ea4340b6257a76685e74176c51dad1b96c
Sep 08 15:52:26.475887 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:26.475822    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8" event={"ID":"ea2d196e-d319-4593-8139-4651c5493784","Type":"ContainerStarted","Data":"b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a"}
Sep 08 15:52:26.476334 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:26.475895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8" event={"ID":"ea2d196e-d319-4593-8139-4651c5493784","Type":"ContainerStarted","Data":"126fcaaf1440ad15c459c9823d17d4ea4340b6257a76685e74176c51dad1b96c"}
Sep 08 15:52:26.497081 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:26.497037    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8" podStartSLOduration=1.49702281 podStartE2EDuration="1.49702281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:52:26.496302894 +0000 UTC m=+2095.571595192" watchObservedRunningTime="2026-09-08 15:52:26.49702281 +0000 UTC m=+2095.572315094"
Sep 08 15:52:28.200522 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:28.200485    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:52:35.721903 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:35.721867    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:35.722455 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:35.721956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:35.740342 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:35.740317    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:36.529984 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:36.529953    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:52:56.268432 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:56.268398    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"]
Sep 08 15:52:58.059081 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:58.059039    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"]
Sep 08 15:52:58.397264 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:52:58.397230    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:07.413068 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:07.413031    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:11.289126 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.289058    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh" podUID="7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" containerName="main" containerID="cri-o://a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c" gracePeriod=45
Sep 08 15:53:11.546275 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.546223    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:53:11.657064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657038    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-home\") pod \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") "
Sep 08 15:53:11.657186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657076    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tls-certs\") pod \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") "
Sep 08 15:53:11.657186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657106    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-model-cache\") pod \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") "
Sep 08 15:53:11.657186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657128    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tmp-dir\") pod \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") "
Sep 08 15:53:11.657186 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657177    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-dshm\") pod \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") "
Sep 08 15:53:11.657390 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkr8l\" (UniqueName: \"kubernetes.io/projected/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-kube-api-access-lkr8l\") pod \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\" (UID: \"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8\") "
Sep 08 15:53:11.657390 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657327    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-home" (OuterVolumeSpecName: "home") pod "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" (UID: "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:11.657390 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657344    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" (UID: "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:11.657554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657440    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" (UID: "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:11.657554 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657537    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:11.657635 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657557    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:11.657635 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.657573    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:11.659246 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.659224    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" (UID: "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:11.659396 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.659380    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-dshm" (OuterVolumeSpecName: "dshm") pod "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" (UID: "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:11.659585 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.659562    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-kube-api-access-lkr8l" (OuterVolumeSpecName: "kube-api-access-lkr8l") pod "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" (UID: "7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8"). InnerVolumeSpecName "kube-api-access-lkr8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:53:11.682668 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.682635    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" containerID="a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c" exitCode=0
Sep 08 15:53:11.682766 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.682684    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"
Sep 08 15:53:11.682766 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.682723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh" event={"ID":"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8","Type":"ContainerDied","Data":"a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c"}
Sep 08 15:53:11.682877 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.682768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh" event={"ID":"7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8","Type":"ContainerDied","Data":"db851a2ac67dfaeb9830027eb5c06ab91d57bcfa6589831e33880c8073f92740"}
Sep 08 15:53:11.682877 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.682790    2575 scope.go:117] "RemoveContainer" containerID="a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c"
Sep 08 15:53:11.691119 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.691104    2575 scope.go:117] "RemoveContainer" containerID="a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c"
Sep 08 15:53:11.691413 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:53:11.691392    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c\": container with ID starting with a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c not found: ID does not exist" containerID="a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c"
Sep 08 15:53:11.691461 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.691423    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c"} err="failed to get container status \"a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c\": rpc error: code = NotFound desc = could not find container \"a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c\": container with ID starting with a1b240813856787fdc9a2603d58f9ec740917415012036d32ad2a6c91403ae1c not found: ID does not exist"
Sep 08 15:53:11.705230 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.705211    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"]
Sep 08 15:53:11.710124 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.710102    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bc1e7aad-kserve-75b454ddb7-hf6gh"]
Sep 08 15:53:11.758258 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.758234    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:11.758341 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.758257    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkr8l\" (UniqueName: \"kubernetes.io/projected/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-kube-api-access-lkr8l\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:11.758341 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:11.758273    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.079677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.079627    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8" podUID="ea2d196e-d319-4593-8139-4651c5493784" containerName="main" containerID="cri-o://b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a" gracePeriod=45
Sep 08 15:53:13.324901 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.324839    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:53:13.372323 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372260    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-dshm\") pod \"ea2d196e-d319-4593-8139-4651c5493784\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") "
Sep 08 15:53:13.372323 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372306    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-home\") pod \"ea2d196e-d319-4593-8139-4651c5493784\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") "
Sep 08 15:53:13.372323 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372323    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-model-cache\") pod \"ea2d196e-d319-4593-8139-4651c5493784\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") "
Sep 08 15:53:13.372558 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372352    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd8km\" (UniqueName: \"kubernetes.io/projected/ea2d196e-d319-4593-8139-4651c5493784-kube-api-access-wd8km\") pod \"ea2d196e-d319-4593-8139-4651c5493784\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") "
Sep 08 15:53:13.372558 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372416    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-tmp-dir\") pod \"ea2d196e-d319-4593-8139-4651c5493784\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") "
Sep 08 15:53:13.372669 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372557    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2d196e-d319-4593-8139-4651c5493784-tls-certs\") pod \"ea2d196e-d319-4593-8139-4651c5493784\" (UID: \"ea2d196e-d319-4593-8139-4651c5493784\") "
Sep 08 15:53:13.372669 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372565    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea2d196e-d319-4593-8139-4651c5493784" (UID: "ea2d196e-d319-4593-8139-4651c5493784"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:13.372669 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372582    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-home" (OuterVolumeSpecName: "home") pod "ea2d196e-d319-4593-8139-4651c5493784" (UID: "ea2d196e-d319-4593-8139-4651c5493784"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:13.372872 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372672    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea2d196e-d319-4593-8139-4651c5493784" (UID: "ea2d196e-d319-4593-8139-4651c5493784"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:13.372943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372896    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.372943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372910    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.372943 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.372920    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.374578 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.374555    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-dshm" (OuterVolumeSpecName: "dshm") pod "ea2d196e-d319-4593-8139-4651c5493784" (UID: "ea2d196e-d319-4593-8139-4651c5493784"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:13.374677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.374588    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2d196e-d319-4593-8139-4651c5493784-kube-api-access-wd8km" (OuterVolumeSpecName: "kube-api-access-wd8km") pod "ea2d196e-d319-4593-8139-4651c5493784" (UID: "ea2d196e-d319-4593-8139-4651c5493784"). InnerVolumeSpecName "kube-api-access-wd8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:53:13.374677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.374601    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d196e-d319-4593-8139-4651c5493784-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea2d196e-d319-4593-8139-4651c5493784" (UID: "ea2d196e-d319-4593-8139-4651c5493784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:13.441131 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.441109    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" path="/var/lib/kubelet/pods/7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8/volumes"
Sep 08 15:53:13.473613 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.473588    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8km\" (UniqueName: \"kubernetes.io/projected/ea2d196e-d319-4593-8139-4651c5493784-kube-api-access-wd8km\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.473613 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.473612    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2d196e-d319-4593-8139-4651c5493784-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.473736 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.473622    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2d196e-d319-4593-8139-4651c5493784-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:13.692931 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.692901    2575 generic.go:356] "Generic (PLEG): container finished" podID="ea2d196e-d319-4593-8139-4651c5493784" containerID="b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a" exitCode=0
Sep 08 15:53:13.693051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.692946    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"
Sep 08 15:53:13.693051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.692978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8" event={"ID":"ea2d196e-d319-4593-8139-4651c5493784","Type":"ContainerDied","Data":"b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a"}
Sep 08 15:53:13.693051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.693016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8" event={"ID":"ea2d196e-d319-4593-8139-4651c5493784","Type":"ContainerDied","Data":"126fcaaf1440ad15c459c9823d17d4ea4340b6257a76685e74176c51dad1b96c"}
Sep 08 15:53:13.693051 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.693035    2575 scope.go:117] "RemoveContainer" containerID="b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a"
Sep 08 15:53:13.701193 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.701173    2575 scope.go:117] "RemoveContainer" containerID="b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a"
Sep 08 15:53:13.701478 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:53:13.701459    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a\": container with ID starting with b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a not found: ID does not exist" containerID="b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a"
Sep 08 15:53:13.701539 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.701485    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a"} err="failed to get container status \"b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a\": rpc error: code = NotFound desc = could not find container \"b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a\": container with ID starting with b161a31a21b00b9e168c2579010b49d0eaa1a55f732d0b7c15f4316bd897fb3a not found: ID does not exist"
Sep 08 15:53:13.713347 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.713315    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"]
Sep 08 15:53:13.717286 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:13.717262    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bc1e7aad-kserve-5fdfc5bfdc-mmcs8"]
Sep 08 15:53:14.815014 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:14.814954    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:15.441046 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:15.441006    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2d196e-d319-4593-8139-4651c5493784" path="/var/lib/kubelet/pods/ea2d196e-d319-4593-8139-4651c5493784/volumes"
Sep 08 15:53:21.629777 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.629737    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"]
Sep 08 15:53:21.630261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.630090    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d196e-d319-4593-8139-4651c5493784" containerName="main"
Sep 08 15:53:21.630261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.630104    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d196e-d319-4593-8139-4651c5493784" containerName="main"
Sep 08 15:53:21.630261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.630115    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" containerName="main"
Sep 08 15:53:21.630261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.630120    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" containerName="main"
Sep 08 15:53:21.630261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.630170    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2d196e-d319-4593-8139-4651c5493784" containerName="main"
Sep 08 15:53:21.630261 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.630178    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1c2bbd-56e8-4d1f-bbec-3eff337c6de8" containerName="main"
Sep 08 15:53:21.634591 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.634567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.637746 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.637714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 15:53:21.639230 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.639203    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:53:21.639369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.639257    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:53:21.639369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.639297    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2xlj4\""
Sep 08 15:53:21.639369 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.639318    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:53:21.645999 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.645975    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"]
Sep 08 15:53:21.737945 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.737918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-metrics-certs\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.738064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.737954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda91688-4fe8-45ff-b338-185873e920c2-openshift-service-ca\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.738064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.737979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-maas-api-tls\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.738064 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.738000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-685xb\" (UniqueName: \"kubernetes.io/projected/dda91688-4fe8-45ff-b338-185873e920c2-kube-api-access-685xb\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.794518 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.794493    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:21.839229 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.839205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-metrics-certs\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.839323 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.839238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda91688-4fe8-45ff-b338-185873e920c2-openshift-service-ca\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.839323 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.839262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-maas-api-tls\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.839438 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.839403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-685xb\" (UniqueName: \"kubernetes.io/projected/dda91688-4fe8-45ff-b338-185873e920c2-kube-api-access-685xb\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.839994 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.839972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda91688-4fe8-45ff-b338-185873e920c2-openshift-service-ca\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.841990 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.841969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-maas-api-tls\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.842074 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.842046    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-metrics-certs\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.848116 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.848094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-685xb\" (UniqueName: \"kubernetes.io/projected/dda91688-4fe8-45ff-b338-185873e920c2-kube-api-access-685xb\") pod \"maas-api-7f8554757b-chlsk\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:21.946876 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:21.946837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:22.218023 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:22.217956    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"]
Sep 08 15:53:22.220307 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:53:22.220278    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda91688_4fe8_45ff_b338_185873e920c2.slice/crio-570c1e8b75de11e0f7327ff4c0f21e75cd5e9dd33135c98f0b952173936ff6fb WatchSource:0}: Error finding container 570c1e8b75de11e0f7327ff4c0f21e75cd5e9dd33135c98f0b952173936ff6fb: Status 404 returned error can't find the container with id 570c1e8b75de11e0f7327ff4c0f21e75cd5e9dd33135c98f0b952173936ff6fb
Sep 08 15:53:22.222032 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:22.222015    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:53:22.735938 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:22.735906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk" event={"ID":"dda91688-4fe8-45ff-b338-185873e920c2","Type":"ContainerStarted","Data":"570c1e8b75de11e0f7327ff4c0f21e75cd5e9dd33135c98f0b952173936ff6fb"}
Sep 08 15:53:23.743602 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:23.743557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk" event={"ID":"dda91688-4fe8-45ff-b338-185873e920c2","Type":"ContainerStarted","Data":"c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a"}
Sep 08 15:53:23.744239 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:23.743693    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:23.763706 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:23.763651    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk" podStartSLOduration=2.056306511 podStartE2EDuration="2.763632943s" totalImagesPullingTime="707.326432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:53:23.761733153 +0000 UTC m=+2152.837025467" watchObservedRunningTime="2026-09-08 15:53:23.763632943 +0000 UTC m=+2152.838925236"
Sep 08 15:53:29.393298 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.393247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:29.481113 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.481082    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"]
Sep 08 15:53:29.481308 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.481278    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk" podUID="dda91688-4fe8-45ff-b338-185873e920c2" containerName="maas-api" containerID="cri-o://c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a" gracePeriod=30
Sep 08 15:53:29.486911 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.486887    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:29.517471 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.517446    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"]
Sep 08 15:53:29.520740 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.520723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.529666 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.529641    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"]
Sep 08 15:53:29.603178 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.603156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7ae7f2-535e-4919-8032-221ecf2a659b-openshift-service-ca\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.603291 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.603193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7ae7f2-535e-4919-8032-221ecf2a659b-maas-api-tls\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.603291 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.603277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7ae7f2-535e-4919-8032-221ecf2a659b-metrics-certs\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.603384 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.603314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mxf\" (UniqueName: \"kubernetes.io/projected/4b7ae7f2-535e-4919-8032-221ecf2a659b-kube-api-access-x9mxf\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.704459 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.704423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7ae7f2-535e-4919-8032-221ecf2a659b-openshift-service-ca\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.704600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.704476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7ae7f2-535e-4919-8032-221ecf2a659b-maas-api-tls\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.704600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.704520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7ae7f2-535e-4919-8032-221ecf2a659b-metrics-certs\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.704600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.704554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mxf\" (UniqueName: \"kubernetes.io/projected/4b7ae7f2-535e-4919-8032-221ecf2a659b-kube-api-access-x9mxf\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.705188 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.705155    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7ae7f2-535e-4919-8032-221ecf2a659b-openshift-service-ca\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.707339 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.707312    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7ae7f2-535e-4919-8032-221ecf2a659b-maas-api-tls\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.707462 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.707362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7ae7f2-535e-4919-8032-221ecf2a659b-metrics-certs\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.713395 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.713372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mxf\" (UniqueName: \"kubernetes.io/projected/4b7ae7f2-535e-4919-8032-221ecf2a659b-kube-api-access-x9mxf\") pod \"maas-api-c5bcdfbdd-hcfs2\" (UID: \"4b7ae7f2-535e-4919-8032-221ecf2a659b\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.731441 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.731419    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:29.775658 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.775630    2575 generic.go:356] "Generic (PLEG): container finished" podID="dda91688-4fe8-45ff-b338-185873e920c2" containerID="c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a" exitCode=0
Sep 08 15:53:29.775803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.775665    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"
Sep 08 15:53:29.775803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.775676    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk" event={"ID":"dda91688-4fe8-45ff-b338-185873e920c2","Type":"ContainerDied","Data":"c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a"}
Sep 08 15:53:29.775803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.775728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-chlsk" event={"ID":"dda91688-4fe8-45ff-b338-185873e920c2","Type":"ContainerDied","Data":"570c1e8b75de11e0f7327ff4c0f21e75cd5e9dd33135c98f0b952173936ff6fb"}
Sep 08 15:53:29.775803 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.775760    2575 scope.go:117] "RemoveContainer" containerID="c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a"
Sep 08 15:53:29.784213 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.784196    2575 scope.go:117] "RemoveContainer" containerID="c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a"
Sep 08 15:53:29.784500 ip-10-0-142-52 kubenswrapper[2575]: E0908 15:53:29.784483    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a\": container with ID starting with c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a not found: ID does not exist" containerID="c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a"
Sep 08 15:53:29.784555 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.784509    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a"} err="failed to get container status \"c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a\": rpc error: code = NotFound desc = could not find container \"c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a\": container with ID starting with c75154700070ef8a6557cf9a0c539fe98d8390bae5f18bcc4142a4573216d25a not found: ID does not exist"
Sep 08 15:53:29.805782 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.805759    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-685xb\" (UniqueName: \"kubernetes.io/projected/dda91688-4fe8-45ff-b338-185873e920c2-kube-api-access-685xb\") pod \"dda91688-4fe8-45ff-b338-185873e920c2\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") "
Sep 08 15:53:29.805905 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.805812    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-metrics-certs\") pod \"dda91688-4fe8-45ff-b338-185873e920c2\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") "
Sep 08 15:53:29.805905 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.805871    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-maas-api-tls\") pod \"dda91688-4fe8-45ff-b338-185873e920c2\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") "
Sep 08 15:53:29.805905 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.805898    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda91688-4fe8-45ff-b338-185873e920c2-openshift-service-ca\") pod \"dda91688-4fe8-45ff-b338-185873e920c2\" (UID: \"dda91688-4fe8-45ff-b338-185873e920c2\") "
Sep 08 15:53:29.806259 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.806235    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda91688-4fe8-45ff-b338-185873e920c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dda91688-4fe8-45ff-b338-185873e920c2" (UID: "dda91688-4fe8-45ff-b338-185873e920c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:53:29.807979 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.807955    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dda91688-4fe8-45ff-b338-185873e920c2" (UID: "dda91688-4fe8-45ff-b338-185873e920c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:29.808081 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.808020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda91688-4fe8-45ff-b338-185873e920c2-kube-api-access-685xb" (OuterVolumeSpecName: "kube-api-access-685xb") pod "dda91688-4fe8-45ff-b338-185873e920c2" (UID: "dda91688-4fe8-45ff-b338-185873e920c2"). InnerVolumeSpecName "kube-api-access-685xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:53:29.808081 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.808046    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dda91688-4fe8-45ff-b338-185873e920c2" (UID: "dda91688-4fe8-45ff-b338-185873e920c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:29.846264 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.846242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:29.907641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.907539    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-685xb\" (UniqueName: \"kubernetes.io/projected/dda91688-4fe8-45ff-b338-185873e920c2-kube-api-access-685xb\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:29.907641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.907577    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:29.907641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.907603    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda91688-4fe8-45ff-b338-185873e920c2-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:29.907641 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:29.907617    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda91688-4fe8-45ff-b338-185873e920c2-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 08 15:53:30.098116 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.098090    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"]
Sep 08 15:53:30.104621 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.104597    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-chlsk"]
Sep 08 15:53:30.120007 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.119984    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"]
Sep 08 15:53:30.121837 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:53:30.121814    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7ae7f2_535e_4919_8032_221ecf2a659b.slice/crio-2a7f5f79b6cd071267bbf05a2d1aafe310ef1ff5e594b94564d0f7a70c807c6f WatchSource:0}: Error finding container 2a7f5f79b6cd071267bbf05a2d1aafe310ef1ff5e594b94564d0f7a70c807c6f: Status 404 returned error can't find the container with id 2a7f5f79b6cd071267bbf05a2d1aafe310ef1ff5e594b94564d0f7a70c807c6f
Sep 08 15:53:30.782625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.782542    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2" event={"ID":"4b7ae7f2-535e-4919-8032-221ecf2a659b","Type":"ContainerStarted","Data":"e0638820fa87777ff6c61d53197d59cf1e4e1ad21450a378f8581bca4c58199a"}
Sep 08 15:53:30.783067 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.782627    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:30.783067 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.782653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2" event={"ID":"4b7ae7f2-535e-4919-8032-221ecf2a659b","Type":"ContainerStarted","Data":"2a7f5f79b6cd071267bbf05a2d1aafe310ef1ff5e594b94564d0f7a70c807c6f"}
Sep 08 15:53:30.801236 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:30.801188    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2" podStartSLOduration=1.3965097819999999 podStartE2EDuration="1.801174673s" totalImagesPullingTime="404.664891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:53:30.800439021 +0000 UTC m=+2159.875731326" watchObservedRunningTime="2026-09-08 15:53:30.801174673 +0000 UTC m=+2159.876466956"
Sep 08 15:53:31.441649 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:31.441605    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda91688-4fe8-45ff-b338-185873e920c2" path="/var/lib/kubelet/pods/dda91688-4fe8-45ff-b338-185873e920c2/volumes"
Sep 08 15:53:36.794289 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:36.794253    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-hcfs2"
Sep 08 15:53:37.518788 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:37.518752    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:44.702792 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:44.702751    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:53:52.599883 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:53:52.599821    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h5268"]
Sep 08 15:54:01.854563 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:01.854529    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-847d47f6c8-w8fgf_8ed1657f-7a61-4f27-a5ca-6671c615f9cc/copy-manifests/0.log"
Sep 08 15:54:01.861581 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:01.861555    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-847d47f6c8-w8fgf_8ed1657f-7a61-4f27-a5ca-6671c615f9cc/manager/0.log"
Sep 08 15:54:02.205665 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:02.205634    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c5bcdfbdd-hcfs2_4b7ae7f2-535e-4919-8032-221ecf2a659b/maas-api/0.log"
Sep 08 15:54:02.439500 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:02.439472    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814690-rqt68_0c6cb92e-3dd3-4801-b487-fed820d147e3/cleanup/0.log"
Sep 08 15:54:02.550534 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:02.550468    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814705-tzcxl_9caed809-64c1-4b8e-b97f-e2d05380257d/cleanup/0.log"
Sep 08 15:54:03.811241 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:03.811217    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-zsh5k_e30b2636-1909-4242-81e0-2be93bd24e21/kuadrant-console-plugin/0.log"
Sep 08 15:54:04.075031 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:04.074950    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-h5268_55cef3ff-3c94-4cac-af00-f8025af07dce/limitador/0.log"
Sep 08 15:54:04.806172 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:04.806138    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d44f46d57-7dn5g_04111a4c-5278-49c3-a692-a3412baf5e08/kube-auth-proxy/0.log"
Sep 08 15:54:05.045798 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:05.045768    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/1.log"
Sep 08 15:54:06.326691 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:06.326649    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-dxfc2_b271884c-a332-4576-ab7e-de755e4c7375/main/0.log"
Sep 08 15:54:06.448837 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:06.448804    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7vnrf_84a66a2f-a79c-4575-9d7d-18e82386cfdf/main/0.log"
Sep 08 15:54:15.136551 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:15.136513    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m66rl_37ac627f-21b3-469b-b8ed-478d372b7f0a/global-pull-secret-syncer/0.log"
Sep 08 15:54:15.227963 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:15.227925    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fmx62_6371911b-a1b5-4a63-a69f-cfdda1648782/konnectivity-agent/0.log"
Sep 08 15:54:15.352615 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:15.352586    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-52.ec2.internal_8f119684184aeb2e23bdcf787e218d99/haproxy/0.log"
Sep 08 15:54:20.742383 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:20.742350    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-zsh5k_e30b2636-1909-4242-81e0-2be93bd24e21/kuadrant-console-plugin/0.log"
Sep 08 15:54:20.870512 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:20.870489    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-h5268_55cef3ff-3c94-4cac-af00-f8025af07dce/limitador/0.log"
Sep 08 15:54:22.878673 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:22.878640    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-6qhkd_3340d218-0e75-47b1-92b2-b257f24b51ce/monitoring-plugin/0.log"
Sep 08 15:54:22.949592 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:22.949552    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fw8v_d77f885d-cd3e-4af3-ad90-adce419f8b25/node-exporter/0.log"
Sep 08 15:54:22.994625 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:22.994601    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fw8v_d77f885d-cd3e-4af3-ad90-adce419f8b25/kube-rbac-proxy/0.log"
Sep 08 15:54:23.023490 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.023469    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fw8v_d77f885d-cd3e-4af3-ad90-adce419f8b25/init-textfile/0.log"
Sep 08 15:54:23.313138 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.313100    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/prometheus/0.log"
Sep 08 15:54:23.334659 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.334637    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/config-reloader/0.log"
Sep 08 15:54:23.365920 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.365889    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/thanos-sidecar/0.log"
Sep 08 15:54:23.391607 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.391591    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/kube-rbac-proxy-web/0.log"
Sep 08 15:54:23.414528 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.414488    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/kube-rbac-proxy/0.log"
Sep 08 15:54:23.436133 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.436117    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/kube-rbac-proxy-thanos/0.log"
Sep 08 15:54:23.468206 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:23.468182    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43f3f747-97a9-4992-8e4a-cc44fd93f4d8/init-config-reloader/0.log"
Sep 08 15:54:25.147820 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.147780    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-sr7vw_48cc9678-ebe1-450d-8fe8-6c5f955298fb/networking-console-plugin/0.log"
Sep 08 15:54:25.605677 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.605603    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"]
Sep 08 15:54:25.605944 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.605931    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda91688-4fe8-45ff-b338-185873e920c2" containerName="maas-api"
Sep 08 15:54:25.605997 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.605946    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda91688-4fe8-45ff-b338-185873e920c2" containerName="maas-api"
Sep 08 15:54:25.606032 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.606012    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda91688-4fe8-45ff-b338-185873e920c2" containerName="maas-api"
Sep 08 15:54:25.609045 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.609029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.611895 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.611878    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-czmnt\"/\"kube-root-ca.crt\""
Sep 08 15:54:25.613355 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.613332    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-czmnt\"/\"default-dockercfg-fqwj2\""
Sep 08 15:54:25.613355 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.613347    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-czmnt\"/\"openshift-service-ca.crt\""
Sep 08 15:54:25.617950 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.617921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"]
Sep 08 15:54:25.706414 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.706394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-lib-modules\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.706516 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.706436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-proc\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.706560 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.706518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994gm\" (UniqueName: \"kubernetes.io/projected/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-kube-api-access-994gm\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.706600 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.706582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-sys\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.706636 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.706612    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-podres\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.807870 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.807828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-podres\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.807987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.807893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-lib-modules\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.807987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.807927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-proc\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.807987 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.807961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994gm\" (UniqueName: \"kubernetes.io/projected/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-kube-api-access-994gm\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.808235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.807989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-podres\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.808235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.808007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-sys\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.808235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.808028    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-proc\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.808235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.808052    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-sys\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.808235 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.808058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-lib-modules\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.818001 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.817971    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-994gm\" (UniqueName: \"kubernetes.io/projected/bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50-kube-api-access-994gm\") pod \"perf-node-gather-daemonset-8z5x7\" (UID: \"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50\") " pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:25.919880 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:25.919840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:26.191670 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:26.191646    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"]
Sep 08 15:54:26.193737 ip-10-0-142-52 kubenswrapper[2575]: W0908 15:54:26.193710    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd5fd7eb_dc6e_4f8a_b505_e1c27a7a8a50.slice/crio-5b3fb83623ec793edffc8355a3b8d97d9d334617c9abd28a12c726c30584d970 WatchSource:0}: Error finding container 5b3fb83623ec793edffc8355a3b8d97d9d334617c9abd28a12c726c30584d970: Status 404 returned error can't find the container with id 5b3fb83623ec793edffc8355a3b8d97d9d334617c9abd28a12c726c30584d970
Sep 08 15:54:27.057522 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:27.057485    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7" event={"ID":"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50","Type":"ContainerStarted","Data":"d666dc3240c38f269dbf916ba6374c789313e2e04770e46f89329ca3945c136e"}
Sep 08 15:54:27.057719 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:27.057566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7" event={"ID":"bd5fd7eb-dc6e-4f8a-b505-e1c27a7a8a50","Type":"ContainerStarted","Data":"5b3fb83623ec793edffc8355a3b8d97d9d334617c9abd28a12c726c30584d970"}
Sep 08 15:54:27.057719 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:27.057604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:27.078157 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:27.078109    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7" podStartSLOduration=2.078093801 podStartE2EDuration="2.078093801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:54:27.077233436 +0000 UTC m=+2216.152525729" watchObservedRunningTime="2026-09-08 15:54:27.078093801 +0000 UTC m=+2216.153386136"
Sep 08 15:54:28.242462 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:28.242435    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v9cbd_379eddb9-5555-40ce-91d2-e88f787f7d27/dns/0.log"
Sep 08 15:54:28.260871 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:28.260831    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v9cbd_379eddb9-5555-40ce-91d2-e88f787f7d27/kube-rbac-proxy/0.log"
Sep 08 15:54:28.280364 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:28.280341    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bc6nc_c094bb8e-9ba5-40ad-97c5-3d236e59f365/dns-node-resolver/0.log"
Sep 08 15:54:28.905784 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:28.905725    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gzb7q_eff82d57-cf61-4995-8dd9-d3da3aa6269d/node-ca/0.log"
Sep 08 15:54:30.220197 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:30.220165    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d44f46d57-7dn5g_04111a4c-5278-49c3-a692-a3412baf5e08/kube-auth-proxy/0.log"
Sep 08 15:54:30.285910 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:30.285885    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/0.log"
Sep 08 15:54:30.304948 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:30.304927    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/1.log"
Sep 08 15:54:31.283826 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:31.283787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lt4mz_a2d7fb1d-6695-45af-ba39-0cb7ed49e6f6/serve-healthcheck-canary/0.log"
Sep 08 15:54:31.927725 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:31.927696    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svxgm_5058d606-9e4a-4d96-8293-23b0245cba81/kube-rbac-proxy/0.log"
Sep 08 15:54:31.954952 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:31.954922    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svxgm_5058d606-9e4a-4d96-8293-23b0245cba81/exporter/0.log"
Sep 08 15:54:31.983331 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:31.983313    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svxgm_5058d606-9e4a-4d96-8293-23b0245cba81/extractor/0.log"
Sep 08 15:54:33.071992 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:33.071963    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-czmnt/perf-node-gather-daemonset-8z5x7"
Sep 08 15:54:35.014660 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:35.014628    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-847d47f6c8-w8fgf_8ed1657f-7a61-4f27-a5ca-6671c615f9cc/manager/0.log"
Sep 08 15:54:35.037270 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:35.037245    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-847d47f6c8-w8fgf_8ed1657f-7a61-4f27-a5ca-6671c615f9cc/copy-manifests/0.log"
Sep 08 15:54:36.488786 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:36.488746    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bc9d6d9f9-m8fwk_202837a5-dc72-49fa-b759-1a3cb0b64eed/manager/0.log"
Sep 08 15:54:36.998715 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:36.998636    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d44f46d57-7dn5g_04111a4c-5278-49c3-a692-a3412baf5e08/kube-auth-proxy/0.log"
Sep 08 15:54:37.045145 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:37.045123    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/0.log"
Sep 08 15:54:37.062911 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:37.062890    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/1.log"
Sep 08 15:54:39.020517 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:39.020484    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d44f46d57-7dn5g_04111a4c-5278-49c3-a692-a3412baf5e08/kube-auth-proxy/0.log"
Sep 08 15:54:39.076422 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:39.076366    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/0.log"
Sep 08 15:54:39.094391 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:39.094362    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6k5qw_7170ec93-6665-4421-8ba3-9d6747ad0146/istio-proxy/1.log"
Sep 08 15:54:42.178374 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:42.178338    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-4t8ck_342f7926-a759-4ae1-a2d8-a58500895217/migrator/0.log"
Sep 08 15:54:42.198118 ip-10-0-142-52 kubenswrapper[2575]: I0908 15:54:42.198093    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-4t8ck_342f7926-a759-4ae1-a2d8-a58500895217/graceful-termination/0.log"